Release 2021.06.09

All tests of `prepare_outtmpl` is now also run on `prepare_filename`

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.08. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.08**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.04.22
+ [debug] yt-dlp version 2021.06.08
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.08. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.08**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.08. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.08**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.08. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.08**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.04.22
+ [debug] yt-dlp version 2021.06.08
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.04.22. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.08. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.04.22**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.08**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/build.yml

@@ -7,13 +7,13 @@ on:
 
 jobs:
   build_unix:
-
     runs-on: ubuntu-latest
 
     outputs:
       ytdlp_version: ${{ steps.bump_version.outputs.ytdlp_version }}
       upload_url: ${{ steps.create_release.outputs.upload_url }}
-      sha2_unix: ${{ steps.sha2_file.outputs.sha2_unix }}
+      sha256_unix: ${{ steps.sha256_file.outputs.sha256_unix }}
+      sha512_unix: ${{ steps.sha512_file.outputs.sha512_unix }}
 
     steps:
     - uses: actions/checkout@v2
@@ -29,7 +29,7 @@ jobs:
     - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlp_version }}"
     - name: Run Make
-      run: make
+      run: make all tar
     - name: Create Release
       id: create_release
       uses: actions/create-release@v1
@@ -53,9 +53,21 @@ jobs:
         asset_path: ./yt-dlp
         asset_name: yt-dlp
         asset_content_type: application/octet-stream
+    - name: Upload Source tar
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ steps.create_release.outputs.upload_url }}
+        asset_path: ./yt-dlp.tar.gz
+        asset_name: yt-dlp.tar.gz
+        asset_content_type: application/gzip
     - name: Get SHA2-256SUMS for yt-dlp
-      id: sha2_file
+      id: sha256_file
-      run: echo "::set-output name=sha2_unix::$(sha256sum yt-dlp | awk '{print $1}')"
+      run: echo "::set-output name=sha256_unix::$(sha256sum yt-dlp | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp
+      id: sha512_file
+      run: echo "::set-output name=sha512_unix::$(sha512sum yt-dlp | awk '{print $1}')"
     - name: Install dependencies for pypi
       env:
         PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
@@ -74,13 +86,12 @@ jobs:
         twine upload dist/*
 
   build_windows:
-
     runs-on: windows-latest
+    needs: build_unix
 
     outputs:
-      sha2_windows: ${{ steps.sha2_file_win.outputs.sha2_windows }}
+      sha256_windows: ${{ steps.sha256_file_win.outputs.sha256_windows }}
-
+      sha512_windows: ${{ steps.sha512_file_win.outputs.sha512_windows }}
-    needs: build_unix
 
     steps:
     - uses: actions/checkout@v2
@@ -110,17 +121,19 @@ jobs:
         asset_name: yt-dlp.exe
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for yt-dlp.exe
-      id: sha2_file_win
+      id: sha256_file_win
-      run: echo "::set-output name=sha2_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
+      run: echo "::set-output name=sha256_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp.exe
+      id: sha512_file_win
+      run: echo "::set-output name=sha512_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())"
 
   build_windows32:
-
     runs-on: windows-latest
+    needs: [build_unix, build_windows]
 
     outputs:
-      sha2_windows32: ${{ steps.sha2_file_win32.outputs.sha2_windows32 }}
+      sha256_windows32: ${{ steps.sha256_file_win32.outputs.sha256_windows32 }}
-
+      sha512_windows32: ${{ steps.sha512_file_win32.outputs.sha512_windows32 }}
-    needs: [build_unix, build_windows]
 
     steps:
     - uses: actions/checkout@v2
@@ -132,7 +145,7 @@ jobs:
     - name: Upgrade pip and enable wheel support
       run: python -m pip install pip==19.1.1 setuptools==43.0.0 wheel==0.33.6
     - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4
+      run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4 pefile==2019.4.18
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
@@ -151,20 +164,28 @@ jobs:
         asset_name: yt-dlp_x86.exe
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for yt-dlp_x86.exe
-      id: sha2_file_win32
+      id: sha256_file_win32
-      run: echo "::set-output name=sha2_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA256).Hash.ToLower())"
+      run: echo "::set-output name=sha256_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp_x86.exe
+      id: sha512_file_win32
+      run: echo "::set-output name=sha512_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA512).Hash.ToLower())"
+
+  finish:
+    runs-on: ubuntu-latest
+    needs: [build_unix, build_windows, build_windows32]
+
+    steps:
     - name: Make SHA2-256SUMS file
       env:
-        SHA2_WINDOWS: ${{ needs.build_windows.outputs.sha2_windows }}
+        SHA256_WINDOWS: ${{ needs.build_windows.outputs.sha256_windows }}
-        SHA2_WINDOWS32: ${{ steps.sha2_file_win32.outputs.sha2_windows32 }}
+        SHA256_WINDOWS32: ${{ needs.build_windows32.outputs.sha256_windows32 }}
-        SHA2_UNIX: ${{ needs.build_unix.outputs.sha2_unix }}
+        SHA256_UNIX: ${{ needs.build_unix.outputs.sha256_unix }}
         YTDLP_VERSION: ${{ needs.build_unix.outputs.ytdlp_version }}
       run: |
-        echo "version:${env:YTDLP_VERSION}" >> SHA2-256SUMS
+        echo "version:${{ env.YTDLP_VERSION }}" >> SHA2-256SUMS
-        echo "yt-dlp.exe:${env:SHA2_WINDOWS}" >> SHA2-256SUMS
+        echo "yt-dlp.exe:${{ env.SHA256_WINDOWS }}" >> SHA2-256SUMS
-        echo "yt-dlp_x86.exe:${env:SHA2_WINDOWS32}" >> SHA2-256SUMS
+        echo "yt-dlp_x86.exe:${{ env.SHA256_WINDOWS32 }}" >> SHA2-256SUMS
-        echo "yt-dlp:${env:SHA2_UNIX}" >> SHA2-256SUMS
+        echo "yt-dlp:${{ env.SHA256_UNIX }}" >> SHA2-256SUMS
-
     - name: Upload 256SUMS file
       id: upload-sums
       uses: actions/upload-release-asset@v1
@@ -175,3 +196,22 @@ jobs:
         asset_path: ./SHA2-256SUMS
         asset_name: SHA2-256SUMS
         asset_content_type: text/plain
+    - name: Make SHA2-512SUMS file
+      env:
+        SHA512_WINDOWS: ${{ needs.build_windows.outputs.sha512_windows }}
+        SHA512_WINDOWS32: ${{ needs.build_windows32.outputs.sha512_windows32 }}
+        SHA512_UNIX: ${{ needs.build_unix.outputs.sha512_unix }}
+      run: |
+        echo "${{ env.SHA512_WINDOWS }}  yt-dlp.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WINDOWS32 }}  yt-dlp_x86.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_UNIX }}  yt-dlp" >> SHA2-512SUMS
+    - name: Upload 512SUMS file
+      id: upload-512sums
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./SHA2-512SUMS
+        asset_name: SHA2-512SUMS
+        asset_content_type: text/plain

.github/workflows/core.yml

@@ -9,53 +9,23 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-18.04]
-        # TODO: python 2.6
+        python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
-        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
-        python-impl: [cpython]
-        ytdl-test-set: [core]
         run-tests-ext: [sh]
         include:
-        # python 3.2 is only available on windows via setup-python
         - os: windows-latest
-          python-version: 3.2
+          python-version: 3.4  # Windows x86 build is still in 3.4
-          python-impl: cpython
-          ytdl-test-set: core
           run-tests-ext: bat
-        # jython
-        - os: ubuntu-latest
-          python-impl: jython
-          ytdl-test-set: core
-          run-tests-ext: sh
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
-      if: ${{ matrix.python-impl == 'cpython' }}
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Set up Java 8
-      if: ${{ matrix.python-impl == 'jython' }}
-      uses: actions/setup-java@v1
-      with:
-        java-version: 8
-    - name: Install Jython
-      if: ${{ matrix.python-impl == 'jython' }}
-      run: |
-        wget https://repo1.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
-        java -jar jython-installer.jar -s -d "$HOME/jython"
-        echo "$HOME/jython/bin" >> $GITHUB_PATH
     - name: Install nose
-      if: ${{ matrix.python-impl != 'jython' }}
       run: pip install nose
-    - name: Install nose (Jython)
-      if: ${{ matrix.python-impl == 'jython' }}
-      # Working around deprecation of support for non-SNI clients at PyPI CDN (see https://status.python.org/incidents/hzmjhqsdjqgb)
-      run: |
-        wget https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
-        pip install nose-1.3.7-py2-none-any.whl
     - name: Run tests
-      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
+      continue-on-error: False
       env:
-        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
+        YTDL_TEST_SET: core
       run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
   # Linter is in quick-test

.github/workflows/download.yml

@@ -9,52 +9,22 @@ jobs:
       fail-fast: true
       matrix:
         os: [ubuntu-18.04]
-        # TODO: python 2.6
+        python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
-        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
-        python-impl: [cpython]
-        ytdl-test-set: [download]
         run-tests-ext: [sh]
         include:
-        # python 3.2 is only available on windows via setup-python
         - os: windows-latest
-          python-version: 3.2
+          python-version: 3.4  # Windows x86 build is still in 3.4
-          python-impl: cpython
-          ytdl-test-set: download
           run-tests-ext: bat
-        # jython - disable for now since it takes too long to complete
-        # - os: ubuntu-latest
-        #   python-impl: jython
-        #   ytdl-test-set: download
-        #   run-tests-ext: sh
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
-      if: ${{ matrix.python-impl == 'cpython' }}
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Set up Java 8
-      if: ${{ matrix.python-impl == 'jython' }}
-      uses: actions/setup-java@v1
-      with:
-        java-version: 8
-    - name: Install Jython
-      if: ${{ matrix.python-impl == 'jython' }}
-      run: |
-        wget https://repo1.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
-        java -jar jython-installer.jar -s -d "$HOME/jython"
-        echo "$HOME/jython/bin" >> $GITHUB_PATH
     - name: Install nose
-      if: ${{ matrix.python-impl != 'jython' }}
       run: pip install nose
-    - name: Install nose (Jython)
-      if: ${{ matrix.python-impl == 'jython' }}
-      # Working around deprecation of support for non-SNI clients at PyPI CDN (see https://status.python.org/incidents/hzmjhqsdjqgb)
-      run: |
-        wget https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
-        pip install nose-1.3.7-py2-none-any.whl
     - name: Run tests
-      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
+      continue-on-error: true
       env:
-        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
+        YTDL_TEST_SET: download
       run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}

.gitignore

@@ -1,3 +1,46 @@
+# Config
+*.conf
+*.spec
+cookies
+cookies.txt
+
+# Downloaded
+*.srt
+*.ttml
+*.sbv
+*.vtt
+*.flv
+*.mp4
+*.m4a
+*.m4v
+*.mp3
+*.3gp
+*.webm
+*.wav
+*.ape
+*.mkv
+*.swf
+*.part
+*.part-*
+*.ytdl
+*.dump
+*.frag
+*.frag.urls
+*.aria2
+*.swp
+*.ogg
+*.opus
+*.info.json
+*.live_chat.json
+*.jpg
+*.png
+*.webp
+*.annotations.xml
+*.description
+
+# Allow config/media files in testdata
+!test/testdata/**
+
 # Python
 *.pyc
 *.pyo
@@ -43,48 +86,6 @@ README.txt
 yt-dlp.zip
 *.exe
 
-# Downloaded
-*.srt
-*.ttml
-*.sbv
-*.vtt
-*.flv
-*.mp4
-*.m4a
-*.m4v
-*.mp3
-*.3gp
-*.webm
-*.wav
-*.ape
-*.mkv
-*.swf
-*.part
-*.part-*
-*.ytdl
-*.dump
-*.frag
-*.frag.urls
-*.aria2
-*.swp
-*.ogg
-*.opus
-*.info.json
-*.live_chat.json
-*.jpg
-*.png
-*.webp
-*.annotations.xml
-*.description
-
-# Config
-*.conf
-*.spec
-cookies
-cookies.txt
-
-
-
 # Text Editor / IDE
 .idea
 *.iml

CONTRIBUTORS

@@ -1,6 +1,7 @@
 pukkandan (owner)
 shirt-dev (collaborator)
 colethedj (collaborator)
+Ashish0804 (collaborator)
 h-h-h-h
 pauldubois98
 nixxo
@@ -20,11 +21,9 @@ FelixFrog
 Zocker1999NET
 nao20010128nao
 kurumigi
-tsukumi
 bbepis
 animelover1984
 Pccode66
-Ashish0804
 RobinD42
 hseg
 DennyDai
@@ -44,3 +43,14 @@ Lamieur
 tsukumijima
 Hadi0609
 b5eff52
+craftingmod
+tpikonen
+tripulse
+king-millez
+alex-gedeon
+hhirtz
+louie-github
+MinePlayersPE
+olifre
+rhsmachine
+nihil-admirari

Changelog.md

@@ -10,7 +10,7 @@
 * Commit to master as `Release <version>`
 * Push to origin/release using `git push origin master:release`
     build task will now run
-* Update version.py using devscripts\update-version.py
+* Update version.py using `devscripts\update-version.py`
 * Run `make issuetemplates`
 * Commit to master as `[version] update :ci skip all`
 * Push to origin/master
@@ -19,16 +19,128 @@
 -->
 
 
-### 2021.05.11
+### 2021.06.09
 
+* Fix bug where `%(field)d` in filename template throws error
+* Improve offset parsing in outtmpl
+* [test] More rigorous tests for `prepare_filename`
+
+### 2021.06.08
+
+* Remove support for obsolete Python versions: Only 3.6+ is now supported
+* Merge youtube-dl: Upto [commit/c2350ca](https://github.com/ytdl-org/youtube-dl/commit/c2350cac243ba1ec1586fe85b0d62d1b700047a2)
+* [hls] Fix decryption for multithreaded downloader
+* [extractor] Fix pre-checking archive for some extractors
+* [extractor] Fix FourCC fallback when parsing ISM by [fstirlitz](https://github.com/fstirlitz)
+* [twitcasting] Add TwitCastingUserIE, TwitCastingLiveIE by [pukkandan](https://github.com/pukkandan), [nao20010128nao](https://github.com/nao20010128nao)
+* [vidio] Add VidioPremierIE and VidioLiveIE by [MinePlayersPE](Https://github.com/MinePlayersPE)
+* [viki] Fix extraction from by [ytdl-org/youtube-dl@59e583f](https://github.com/ytdl-org/youtube-dl/commit/59e583f7e8530ca92776c866897d895c072e2a82)
+* [youtube] Support shorts URL
+* [zoom] Extract transcripts as subtitles
+* Add field `original_url` with the user-inputted URL
+* Fix and refactor `prepare_outtmpl`
+* Make more fields available for `--print` when used with `--flat-playlist`
+* [utils] Generalize `traverse_dict` to `traverse_obj`
+* [downloader/ffmpeg] Hide FFmpeg banner unless in verbose mode by [fstirlitz](https://github.com/fstirlitz)
+* [build] Release `yt-dlp.tar.gz`
+* [build,update] Add GNU-style SHA512 and prepare updater for simlar SHA256 by [nihil-admirari](https://github.com/nihil-admirari)
+* [pyinst] Show Python version in exe metadata by [nihil-admirari](https://github.com/nihil-admirari)
+* [docs] Improve documentation of dependencies
+* [cleanup] Mark unused files
+* [cleanup] Point all shebang to `python3` by [fstirlitz](https://github.com/fstirlitz)
+* [cleanup] Remove duplicate file `trovolive.py`
+
+
+### 2021.06.01
+
+* Merge youtube-dl: Upto [commit/d495292](https://github.com/ytdl-org/youtube-dl/commit/d495292852b6c2f1bd58bc2141ff2b0265c952cf)
+* Pre-check archive and filters during playlist extraction
+* Handle Basic Auth `user:pass` in URLs by [hhirtz](https://github.com/hhirtz) and [pukkandan](https://github.com/pukkandan)
+* [archiveorg] Add YoutubeWebArchiveIE by [colethedj](https://github.com/colethedj) and [alex-gedeon](https://github.com/alex-gedeon)
+* [fancode] Add extractor by [rhsmachine](https://github.com/rhsmachine)
+* [patreon] Support vimeo embeds by [rhsmachine](https://github.com/rhsmachine)
+* [Saitosan] Add new extractor by [llacb47](https://github.com/llacb47)
+* [ShemarooMe] Add extractor by [Ashish0804](https://github.com/Ashish0804) and [pukkandan](https://github.com/pukkandan)
+* [telemundo] Add extractor by [king-millez](https://github.com/king-millez)
+* [SonyLIV] Add SonyLIVSeriesIE and subtitle support by [Ashish0804](https://github.com/Ashish0804)
+* [Hotstar] Add HotStarSeriesIE by [Ashish0804](https://github.com/Ashish0804)
+* [Voot] Add VootSeriesIE by [Ashish0804](https://github.com/Ashish0804)
+* [vidio] Support login and premium videos by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [fragment] When using `-N`, do not keep the fragment content in memory
+* [ffmpeg] Download and merge in a single step if possible
+* [ThumbnailsConvertor] Support conversion to `png` and make it the default by [louie-github](https://github.com/louie-github)
+* [VideoConvertor] Generalize with remuxer and allow conditional recoding
+* [EmbedThumbnail] Embed in `mp4`/`m4a` using mutagen by [tripulse](https://github.com/tripulse) and [pukkandan](https://github.com/pukkandan)
+* [EmbedThumbnail] Embed if any thumbnail was downloaded, not just the best
+* [EmbedThumbnail] Correctly escape filename
+* [update] replace self without launching a subprocess in windows
+* [update] Block further update for unsupported systems
+* Refactor `__process_playlist` by creating `LazyList`
+* Write messages to `stderr` when both `quiet` and `verbose`
+* Sanitize and sort playlist thumbnails
+* Remove `None` values from `info.json`
+* [extractor] Always prefer native hls downloader by default
+* [extractor] Skip subtitles without URI in m3u8 manifests by [hheimbuerger](https://github.com/hheimbuerger)
+* [extractor] Functions to parse `socket.io` response as `json` by [pukkandan](https://github.com/pukkandan) and [llacb47](https://github.com/llacb47)
+* [extractor] Allow `note=False` when extracting manifests
+* [utils] Escape URLs in `sanitized_Request`, not `sanitize_url`
+* [hls] Disable external downloader for `webtt`
+* [youtube] `/live` URLs should raise error if channel is not live
+* [youtube] Bug fixes
+* [zee5] Fix m3u8 formats' extension
+* [ard] Allow URLs without `-` before id by [olifre](https://github.com/olifre)
+* [cleanup] `YoutubeDL._match_entry`
+* [cleanup] Refactor updater
+* [cleanup] Refactor ffmpeg convertors
+* [cleanup] setup.py
+
+
+### 2021.05.20
+
+* **Youtube improvements**: 
+    * Support youtube music `MP`, `VL` and `browse` pages
+    * Extract more formats for youtube music by [craftingmod](https://github.com/craftingmod), [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+    * Extract multiple subtitles in same language by [pukkandan](https://github.com/pukkandan) and [tpikonen](https://github.com/tpikonen)
+    * Redirect channels that doesn't have a `videos` tab to their `UU` playlists
+    * Support in-channel search
+    * Sort audio-only formats correctly
+    * Always extract `maxresdefault` thumbnail
+    * Extract audio language
+    * Add subtitle language names by [nixxo](https://github.com/nixxo) and [tpikonen](https://github.com/tpikonen)
+    * Show alerts only from the final webpage
+    * Add `html5=1` param to `get_video_info` page requests by [colethedj](https://github.com/colethedj)
+    * Better message when login required
+* **Add option `--print`**: to print any field/template
+    * Deprecates: `--get-description`, `--get-duration`, `--get-filename`, `--get-format`, `--get-id`, `--get-thumbnail`, `--get-title`, `--get-url`
+* Field `additional_urls` to download additional videos from metadata using [`--parse-metadata`](https://github.com/yt-dlp/yt-dlp#modifying-metadata)
+* Merge youtube-dl: Upto [commit/dfbbe29](https://github.com/ytdl-org/youtube-dl/commit/dfbbe2902fc67f0f93ee47a8077c148055c67a9b)
+* Write thumbnail of playlist and add `pl_thumbnail` outtmpl key
+* [embedthumbnail] Add `flac` support and refactor `mutagen` code by [pukkandan](https://github.com/pukkandan) and [tripulse](https://github.com/tripulse)
+* [audius:artist] Add extractor by [king-millez](https://github.com/king-millez)
+* [parlview] Add extractor by [king-millez](https://github.com/king-millez)
+* [tenplay] Fix extractor by [king-millez](https://github.com/king-millez)
+* [rmcdecouverte] Generalize `_VALID_URL`
+* Add compat-option `no-attach-infojson`
+* Add field `name` for subtitles
+* Ensure `post_extract` and `pre_process` only run once
+* Fix `--check-formats` when there is network error
+* Standardize `write_debug` and `get_param`
+* [options] Alias `--write-comments`, `--no-write-comments`
+* [options] Refactor callbacks
+* [test:download] Only extract enough videos for `playlist_mincount`
+* [extractor] bugfix for when `compat_opts` is not given
+* [build] Fix x86 build by [shirt](https://github.com/shirt-dev)
+* [cleanup] code formatting, youtube tests and readme
+
+### 2021.05.11
 * **Deprecate support for python versions < 3.6**
-* **Subtitle extraction from manifests** by [fstirlitz](https://github.com/fstirlitz). See [be6202f12b97858b9d716e608394b51065d0419f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
+* **Subtitle extraction from manifests** by [fstirlitz](https://github.com/fstirlitz). See [be6202f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
 * **Improve output template:**
     * Allow slicing lists/strings using `field.start:end:step`
     * A field can also be used as offset like `field1+num+field2`
     * A default value can be given using `field|default`
     * Prevent invalid fields from causing errors
-* Merge youtube-dl: Upto [commit/a726009](https://github.com/ytdl-org/youtube-dl/commit/a7260099873acc6dc7d76cafad2f6b139087afd0)
+* **Merge youtube-dl**: Upto [commit/a726009](https://github.com/ytdl-org/youtube-dl/commit/a7260099873acc6dc7d76cafad2f6b139087afd0)
 * **Remove options** `-l`, `-t`, `-A` completely and disable `--auto-number`, `--title`, `--literal`, `--id`
 * [Plugins] Prioritize plugins over standard extractors and prevent plugins from overwriting the standard extractor classes
 * [downloader] Fix `quiet` and `to_stderr`
@@ -63,7 +175,7 @@
 * [zee5] Fix py2 compatibility
 * Fix `playlist_index` and add `playlist_autonumber`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details
 * Add experimental option `--check-formats` to test the URLs before format selection
-* Option `--compat-options` to revert some of yt-dlp's changes
+* Option `--compat-options` to revert [some of yt-dlp's changes](https://github.com/yt-dlp/yt-dlp#differences-in-default-behavior)
     * Deprecates `--list-formats-as-table`, `--list-formats-old`
 * Fix number of digits in `%(playlist_index)s`
 * Fix case sensitivity of format selector
@@ -320,7 +432,7 @@
 
 ### 2021.02.15
 * Merge youtube-dl: Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
-* [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumi](https://github.com/tsukumi), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
+* [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumijima](https://github.com/tsukumijima), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
 * Fix HLS AES-128 with multiple keys in external downloaders by [shirt](https://github.com/shirt-dev)
 * [youtube_live_chat] Fix by using POST API by [siikamiika](https://github.com/siikamiika)
 * [rumble] Add support for video page

Makefile

@@ -25,6 +25,7 @@ completion-zsh: completions/zsh/_yt-dlp
 lazy-extractors: yt_dlp/extractor/lazy_extractors.py
 
 PREFIX ?= /usr/local
+DESTDIR ?= .
 BINDIR ?= $(PREFIX)/bin
 MANDIR ?= $(PREFIX)/man
 SHAREDIR ?= $(PREFIX)/share

README.md

@@ -22,8 +22,8 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
 * [NEW FEATURES](#new-features)
     * [Differences in default behavior](#differences-in-default-behavior)
 * [INSTALLATION](#installation)
-    * [Dependencies](#dependencies)
     * [Update](#update)
+    * [Dependencies](#dependencies)
     * [Compile](#compile)
 * [USAGE AND OPTIONS](#usage-and-options)
     * [General Options](#general-options)
@@ -66,15 +66,17 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* **Merged with youtube-dl [commit/a726009](https://github.com/ytdl-org/youtube-dl/commit/a7260099873acc6dc7d76cafad2f6b139087afd0)**: (v2021.04.26) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl [commit/c2350ca](https://github.com/ytdl-org/youtube-dl/commit/c2350cac243ba1ec1586fe85b0d62d1b700047a2)**: (v2021.06.06) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 
-* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
+* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--write-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
 
 * **Youtube improvements**:
-    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works and supports downloading multiple pages of content
+    * All Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) supports downloading multiple pages of content
-    * Youtube search (`ytsearch:`, `ytsearchdate:`) along with Search URLs work
+    * Search (`ytsearch:`, `ytsearchdate:`), search URLs and in-channel search works
-    * Youtube mixes supports downloading multiple pages of content
+    * Mixes supports downloading multiple pages of content
     * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
+    * `255kbps` audio is extracted from youtube music if premium cookies are given
+    * Youtube music Albums, channels etc can be downloaded
 
 * **Split video by chapters**: Videos can be split into multiple files based on chapters using `--split-chapters`
 
@@ -82,11 +84,11 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats
 
-* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive
 
-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon
 
-* **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [be6202f12b97858b9d716e608394b51065d0419f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
+* **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [commit/be6202f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
 
 * **Multiple paths and output templates**: You can give different [output templates](#output-template) and download paths for different types of files. You can also set a temporary path where intermediary files are downloaded to using `--paths` (`-P`)
 
@@ -114,24 +116,26 @@ If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the
 
 Some of yt-dlp's default options are different from that of youtube-dl and youtube-dlc.
 
-1. The options `--id`, `--auto-number` (`-A`), `--title` (`-t`) and `--literal` (`-l`), no longer work. See [removed options](#Removed) for details
+* The options `--id`, `--auto-number` (`-A`), `--title` (`-t`) and `--literal` (`-l`), no longer work. See [removed options](#Removed) for details
-1. `avconv` is not supported as as an alternative to `ffmpeg`
+* `avconv` is not supported as as an alternative to `ffmpeg`
-1. The default [output template](#output-template) is `%(title)s [%(id)s].%(ext)s`. There is no real reason for this change. This was changed before yt-dlp was ever made public and now there are no plans to change it back to `%(title)s.%(id)s.%(ext)s`. Instead, you may use `--compat-options filename`
+* The default [output template](#output-template) is `%(title)s [%(id)s].%(ext)s`. There is no real reason for this change. This was changed before yt-dlp was ever made public and now there are no plans to change it back to `%(title)s.%(id)s.%(ext)s`. Instead, you may use `--compat-options filename`
-1. The default [format sorting](sorting-formats) is different from youtube-dl and prefers higher resolution and better codecs rather than higher bitrates. You can use the `--format-sort` option to change this to any order you prefer, or use `--compat-options format-sort` to use youtube-dl's sorting order
+* The default [format sorting](sorting-formats) is different from youtube-dl and prefers higher resolution and better codecs rather than higher bitrates. You can use the `--format-sort` option to change this to any order you prefer, or use `--compat-options format-sort` to use youtube-dl's sorting order
-1. The default format selector is `bv*+ba/b`. This means that if a combined video + audio format that is better than the best video-only format is found, the former will be prefered. Use `-f bv+ba/b` or `--compat-options format-spec` to revert this
+* The default format selector is `bv*+ba/b`. This means that if a combined video + audio format that is better than the best video-only format is found, the former will be prefered. Use `-f bv+ba/b` or `--compat-options format-spec` to revert this
-1. Unlike youtube-dlc, yt-dlp does not allow merging multiple audio/video streams into one file by default (since this conflicts with the use of `-f bv*+ba`). If needed, this feature must be enabled using `--audio-multistreams` and `--video-multistreams`. You can also use `--compat-options multistreams` to enable both
+* Unlike youtube-dlc, yt-dlp does not allow merging multiple audio/video streams into one file by default (since this conflicts with the use of `-f bv*+ba`). If needed, this feature must be enabled using `--audio-multistreams` and `--video-multistreams`. You can also use `--compat-options multistreams` to enable both
-1. `--ignore-errors` is enabled by default. Use `--abort-on-error` or `--compat-options abort-on-error` to abort on errors instead
+* `--ignore-errors` is enabled by default. Use `--abort-on-error` or `--compat-options abort-on-error` to abort on errors instead
-1. When writing metadata files such as thumbnails, description or infojson, the same information (if available) is also written for playlists. Use `--no-write-playlist-metafiles` or `--compat-options no-playlist-metafiles` to not write these files
+* When writing metadata files such as thumbnails, description or infojson, the same information (if available) is also written for playlists. Use `--no-write-playlist-metafiles` or `--compat-options no-playlist-metafiles` to not write these files
-1. `playlist_index` behaves differently when used with options like `--playlist-reverse` and `--playlist-items`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details. You can use `--compat-options playlist-index` if you want to keep the earlier behavior
+* `--add-metadata` attaches the `infojson` to `mkv` files in addition to writing the metadata when used with `--write-infojson`. Use `--compat-options no-attach-info-json` to revert this
-1. The output of `-F` is listed in a new format. Use `--compat-options list-formats` to revert this
+* `playlist_index` behaves differently when used with options like `--playlist-reverse` and `--playlist-items`. See [#302](https://github.com/yt-dlp/yt-dlp/issues/302) for details. You can use `--compat-options playlist-index` if you want to keep the earlier behavior
-1. Youtube live chat (if available) is considered as a subtitle. Use `--sub-langs all,-live_chat` to download all subtitles except live chat. You can also use `--compat-options no-live-chat` to prevent live chat from downloading
+* The output of `-F` is listed in a new format. Use `--compat-options list-formats` to revert this
-1. Youtube channel URLs are automatically redirected to `/video`. Either append a `/featured` to the URL or use `--compat-options no-youtube-channel-redirect` to download only the videos in the home page
+* Youtube live chat (if available) is considered as a subtitle. Use `--sub-langs all,-live_chat` to download all subtitles except live chat. You can also use `--compat-options no-live-chat` to prevent live chat from downloading
-1. Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
+* Youtube channel URLs are automatically redirected to `/video`. Append a `/featured` to the URL to download only the videos in the home page. If the channel does not have a videos tab, we try to download the equivalent `UU` playlist instead. Also, `/live` URLs raise an error if there are no live videos instead of silently downloading the entire channel. You may use `--compat-options no-youtube-channel-redirect` to revert all these redirections
+* Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
+* If `ffmpeg` is used as the downloader, the downloading and merging of formats happen in a single step when possible. Use `--compat-options no-direct-merge` to revert this
 
 For ease of use, a few more compat options are available:
-1. `--compat-options all` = Use all compat options
+* `--compat-options all`: Use all compat options
-1. `--compat-options youtube-dl` = `--compat-options all,-multistreams`
+* `--compat-options youtube-dl`: Same as `--compat-options all,-multistreams`
-1. `--compat-options youtube-dlc` = `--compat-options all,-no-live-chat,-no-youtube-channel-redirect`
+* `--compat-options youtube-dlc`: Same as `--compat-options all,-no-live-chat,-no-youtube-channel-redirect`
 
 
 # INSTALLATION
@@ -162,17 +166,31 @@ sudo aria2c https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o
 sudo chmod a+rx /usr/local/bin/yt-dlp
 ```
 
-### DEPENDENCIES
-Python versions 3.6+ (CPython and PyPy) are officially supported. Other versions and implementations may or maynot work correctly.
-
-On windows, [Microsoft Visual C++ 2010 Redistributable Package (x86)](https://www.microsoft.com/en-us/download/details.aspx?id=26999) is also necessary to run yt-dlp. You probably already have this, but if the executable throws an error due to missing `MSVCR100.dll` you need to install it.
-
-Although there are no other required dependencies, `ffmpeg` and `ffprobe` are highly recommended. Other optional dependencies are `sponskrub`, `AtomicParsley`, `mutagen`, `pycryptodome` and any of the supported external downloaders. Note that the windows releases are already built with the python interpreter, mutagen and pycryptodome included.
-
 ### UPDATE
 You can use `yt-dlp -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.
 
+### DEPENDENCIES
+Python versions 3.6+ (CPython and PyPy) are supported. Other versions and implementations may or may not work correctly.
+
+<!-- https://www.microsoft.com/en-us/download/details.aspx?id=26999 -->
+On windows, [Microsoft Visual C++ 2010 SP1 Redistributable Package (x86)](https://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe) is also necessary to run yt-dlp. You probably already have this, but if the executable throws an error due to missing `MSVCR100.dll` you need to install it manually.
+
+While all the other dependancies are optional, `ffmpeg` and `ffprobe` are highly recommended
+* [**ffmpeg** and **ffprobe**](https://www.ffmpeg.org) - Required for [merging seperate video and audio files](#format-selection) as well as for various [post-processing](#post-processing-options) tasks. Licence [depends on the build](https://www.ffmpeg.org/legal.html)
+* [**sponskrub**](https://github.com/faissaloo/SponSkrub) - For using the [sponskrub options](#sponskrub-sponsorblock-options). Licenced under [GPLv3+](https://github.com/faissaloo/SponSkrub/blob/master/LICENCE.md)
+* [**mutagen**](https://github.com/quodlibet/mutagen) - For embedding thumbnail in certain formats. Licenced under [GPLv2+](https://github.com/quodlibet/mutagen/blob/master/COPYING)
+* [**pycryptodome**](https://github.com/Legrandin/pycryptodome) - For decrypting various data. Licenced under [BSD2](https://github.com/Legrandin/pycryptodome/blob/master/LICENSE.rst)
+* [**AtomicParsley**](https://github.com/wez/atomicparsley) - For embedding thumbnail in mp4/m4a if mutagen is not present. Licenced under [GPLv2+](https://github.com/wez/atomicparsley/blob/master/COPYING)
+* [**rtmpdump**](http://rtmpdump.mplayerhq.hu) - For downloading `rtmp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](http://rtmpdump.mplayerhq.hu)
+* [**mplayer**](http://mplayerhq.hu/design7/info.html) or [**mpv**](https://mpv.io) - For downloading `rstp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](https://github.com/mpv-player/mpv/blob/master/Copyright)
+* [**phantomjs**](https://github.com/ariya/phantomjs) - Used in extractors where javascript needs to be run. Licenced under [BSD3](https://github.com/ariya/phantomjs/blob/master/LICENSE.BSD)
+* Any external downloader that you want to use with `--downloader`
+
+To use or redistribute the dependencies, you must agree to their respective licensing terms.
+
+Note that the windows releases are already built with the python interpreter, mutagen and pycryptodome included.
+
 ### COMPILE
 
 **For Windows**:
@@ -482,10 +500,13 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      could still contain some personal
                                      information (default)
     --no-clean-infojson              Write all fields to the infojson
-    --get-comments                   Retrieve video comments to be placed in the
+    --write-comments                 Retrieve video comments to be placed in the
-                                     .info.json file. The comments are fetched
+                                     infojson. The comments are fetched even
-                                     even without this option if the extraction
+                                     without this option if the extraction is
-                                     is known to be quick
+                                     known to be quick (Alias: --get-comments)
+    --no-write-comments              Do not retrieve video comments unless the
+                                     extraction is known to be quick
+                                     (Alias: --no-get-comments)
     --load-info-json FILE            JSON file containing the video information
                                      (created with the "--write-info-json"
                                      option)
@@ -493,13 +514,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      jar in
     --no-cookies                     Do not read/dump cookies (default)
     --cache-dir DIR                  Location in the filesystem where youtube-dl
-                                     can store some downloaded information
+                                     can store some downloaded information (such
-                                     permanently. By default
+                                     as client ids and signatures) permanently.
-                                     $XDG_CACHE_HOME/youtube-dl or
+                                     By default $XDG_CACHE_HOME/youtube-dl or
-                                     ~/.cache/youtube-dl . At the moment, only
+                                     ~/.cache/youtube-dl
-                                     YouTube player files (for videos with
-                                     obfuscated signatures) are cached, but that
-                                     may change
     --no-cache-dir                   Disable filesystem caching
     --rm-cache-dir                   Delete all filesystem cache files
 
@@ -533,14 +551,9 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      formats are found (default)
     --skip-download                  Do not download the video but write all
                                      related files (Alias: --no-download)
-    -g, --get-url                    Simulate, quiet but print URL
+    -O, --print TEMPLATE             Simulate, quiet but print the given fields.
-    -e, --get-title                  Simulate, quiet but print title
+                                     Either a field name or similar formatting
-    --get-id                         Simulate, quiet but print id
+                                     as the output template can be used
-    --get-thumbnail                  Simulate, quiet but print thumbnail URL
-    --get-description                Simulate, quiet but print video description
-    --get-duration                   Simulate, quiet but print video length
-    --get-filename                   Simulate, quiet but print output filename
-    --get-format                     Simulate, quiet but print output format
     -j, --dump-json                  Simulate, quiet but print JSON information.
                                      See "OUTPUT TEMPLATE" for a description of
                                      available keys
@@ -669,10 +682,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
 ## Post-Processing Options:
     -x, --extract-audio              Convert video files to audio-only files
                                      (requires ffmpeg and ffprobe)
-    --audio-format FORMAT            Specify audio format: "best", "aac",
+    --audio-format FORMAT            Specify audio format to convert the audio
-                                     "flac", "mp3", "m4a", "opus", "vorbis", or
+                                     to when -x is used. Currently supported
-                                     "wav"; "best" by default; No effect without
+                                     formats are: best (default) or one of
-                                     -x
+                                     aac|flac|mp3|m4a|opus|vorbis|wav
     --audio-quality QUALITY          Specify ffmpeg audio quality, insert a
                                      value between 0 (better) and 9 (worse) for
                                      VBR or a specific bitrate like 128K
@@ -682,12 +695,12 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      |webm|mov|avi|mp3|mka|m4a|ogg|opus). If
                                      target container does not support the
                                      video/audio codec, remuxing will fail. You
-                                     can specify multiple rules; eg.
+                                     can specify multiple rules; Eg.
                                      "aac>m4a/mov>mp4/mkv" will remux aac to
                                      m4a, mov to mp4 and anything else to mkv.
     --recode-video FORMAT            Re-encode the video into another format if
-                                     re-encoding is necessary. The supported
+                                     re-encoding is necessary. The syntax and
-                                     formats are the same as --remux-video
+                                     supported formats are the same as --remux-video
     --postprocessor-args NAME:ARGS   Give these arguments to the postprocessors.
                                      Specify the postprocessor/executable name
                                      and the arguments separated by a colon ":"
@@ -748,10 +761,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      fields are passed, "%(filepath)s" is
                                      appended to the end of the command
     --convert-subs FORMAT            Convert the subtitles to another format
-                                     (currently supported: srt|ass|vtt|lrc)
+                                     (currently supported: srt|vtt|ass|lrc)
                                      (Alias: --convert-subtitles)
     --convert-thumbnails FORMAT      Convert the thumbnails to another format
-                                     (currently supported: jpg)
+                                     (currently supported: jpg|png)
     --split-chapters                 Split video into multiple files based on
                                      internal chapters. The "chapter:" prefix
                                      can be used with "--paths" and "--output"
@@ -822,7 +835,7 @@ You can configure yt-dlp by placing any supported command line option to a confi
     * `~/yt-dlp.conf.txt`
 
     Note that `~` points to `C:\Users\<user name>` on windows. Also, `%XDG_CONFIG_HOME%` defaults to `~/.config` if undefined
-1. **System Configuration**: `/etc/yt-dlp.conf` or `/etc/yt-dlp.conf`
+1. **System Configuration**: `/etc/yt-dlp.conf`
 
 For example, with the following configuration file yt-dlp will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory:
 ```
@@ -889,7 +902,7 @@ To summarize, the general syntax for a field is:
 %(name[.keys][addition][>strf][|default])[flags][width][.precision][length]type
 ```
 
-Additionally, you can set different output templates for the various metadata files separately from the general output template by specifying the type of file followed by the template separated by a colon `:`. The different file types supported are `subtitle`, `thumbnail`, `description`, `annotation`, `infojson`, `pl_description`, `pl_infojson`, `chapter`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.
+Additionally, you can set different output templates for the various metadata files separately from the general output template by specifying the type of file followed by the template separated by a colon `:`. The different file types supported are `subtitle`, `thumbnail`, `description`, `annotation`, `infojson`, `pl_thumbnail`, `pl_description`, `pl_infojson`, `chapter`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video.
 
 The available fields are:
 
@@ -911,7 +924,7 @@ The available fields are:
  - `channel_id` (string): Id of the channel
  - `location` (string): Physical location where the video was filmed
  - `duration` (numeric): Length of the video in seconds
- - `duration_string` (string): Length of the video (HH-mm-ss)
+ - `duration_string` (string): Length of the video (HH:mm:ss)
  - `view_count` (numeric): How many users have watched the video on the platform
  - `like_count` (numeric): Number of positive ratings of the video
  - `dislike_count` (numeric): Number of negative ratings of the video
@@ -952,6 +965,8 @@ The available fields are:
  - `playlist_title` (string): Playlist title
  - `playlist_uploader` (string): Full name of the playlist uploader
  - `playlist_uploader_id` (string): Nickname or id of the playlist uploader
+ - `webpage_url` (string): A URL to the video webpage which if given to yt-dlp should allow to get the same result again
+ - `original_url` (string): The URL given by the user (or same as `webpage_url` for playlist entries)
 
 Available for the video that belongs to some logical chapter or section:
 
@@ -989,6 +1004,11 @@ Available for `chapter:` prefix when using `--split-chapters` for videos with in
  - `section_start` (numeric): Start time of the chapter in seconds
  - `section_end` (numeric): End time of the chapter in seconds
 
+Available only when used in `--print`:
+
+ - `urls` (string): The URLs of all requested formats, one in each line
+ - `filename` (string): Name of the video file. Note that the actual filename may be different due to post-processing. Use `--exec echo` to get the name after all postprocessing is complete
+
 Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
 
 For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `yt-dlp test video` and id `BaW_jenozKcj`, this will result in a `yt-dlp test video-BaW_jenozKcj.mp4` file created in the current directory.
@@ -1278,7 +1298,9 @@ The metadata obtained the the extractors can be modified by using `--parse-metad
 
 Note that any field created by this can be used in the [output template](#output-template) and will also affect the media file's metadata added when using `--add-metadata`.
 
-You can also use this to change only the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. You can use this to set a different "description" and "synopsis", for example.
+This option also has a few special uses:
+* You can use this to change the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. You can use this to set a different "description" and "synopsis", for example
+* You can download an additional URL based on the metadata of the currently downloaded video. To do this, set the field `additional_urls` to the URL that you want to download. Eg: `--parse-metadata "description:(?P<additional_urls>https?://www\.vimeo\.com/\d+)` will download the first vimeo video found in the description
 
 ## Modifying metadata examples
 
@@ -1312,6 +1334,14 @@ These are all the deprecated options and the current alternative to achieve the
 #### Not recommended
 While these options still work, their use is not recommended since there are other alternatives to achieve the same
 
+    --get-description                --print description
+    --get-duration                   --print duration_string
+    --get-filename                   --print filename
+    --get-format                     --print format
+    --get-id                         --print id
+    --get-thumbnail                  --print thumbnail
+    -e, --get-title                  --print title
+    -g, --get-url                    --print urls
     --all-formats                    -f all
     --all-subs                       --sub-langs all --write-subs
     --autonumber-size NUMBER         Use string formatting. Eg: %(autonumber)03d
@@ -1323,6 +1353,7 @@ While these options still work, their use is not recommended since there are oth
     --list-formats-as-table          --compat-options -list-formats [Default] (Alias: --no-list-formats-old)
     --sponskrub-args ARGS            --ppa "sponskrub:ARGS"
     --test                           Used by developers for testing extractors. Not intended for the end user
+    --youtube-print-sig-code         Used for testing youtube signatures
 
 
 #### Old aliases
@@ -1353,7 +1384,6 @@ These options may no longer work as intended
     --no-call-home                   Default
     --include-ads                    No longer supported
     --no-include-ads                 Default
-    --youtube-print-sig-code         No longer supported
 
 #### Removed
 These options were deprecated since 2014 and have now been entirely removed

devscripts/bash-completion.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import os

devscripts/buildserver.py

@@ -1,3 +1,5 @@
+# UNUSED
+
 #!/usr/bin/python3
 
 import argparse

devscripts/check-porn.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 """

devscripts/create-github-release.py

@@ -1,6 +1,6 @@
 # Unused
 
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/fish-completion.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import optparse

devscripts/generate_aes_testdata.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import codecs

devscripts/gh-pages.unused/update-copyright.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import with_statement, unicode_literals

devscripts/lazy_load_template.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

devscripts/make_contributing.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # import io

devscripts/make_issue_template.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/make_lazy_extractors.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals, print_function
 
 from inspect import getsource

devscripts/make_readme.py

@@ -1,3 +1,8 @@
+#!/usr/bin/env python3
+
+# yt-dlp --help | make_readme.py
+# This must be run in a console of correct width
+
 from __future__ import unicode_literals
 
 import io

devscripts/make_supportedsites.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/prepare_manpage.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/release.sh

@@ -1,4 +1,5 @@
 # Unused
+
 #!/bin/bash
 
 # IMPORTANT: the following assumptions are made

devscripts/show-downloads-statistics.py

@@ -1,6 +1,6 @@
 # Unused
 
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import itertools

devscripts/update-version.py

@@ -1,4 +1,6 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals
+
 from datetime import datetime
 # import urllib.request
 

devscripts/wine-py2exe.sh

@@ -1,3 +1,5 @@
+# UNUSED
+
 #!/bin/bash
 
 # Run with as parameter a setup.py that works in the current directory

devscripts/zsh-completion.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import os

pyinst.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals
@@ -17,7 +17,7 @@ assert arch in ('32', '64')
 print('Building %sbit version' % arch)
 _x86 = '_x86' if arch == '32' else ''
 
-FILE_DESCRIPTION = 'Media Downloader%s' % (' (32 Bit)' if _x86 else '')
+FILE_DESCRIPTION = 'yt-dlp%s' % (' (32 Bit)' if _x86 else '')
 
 # root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 # print('Changing working directory to %s' % root_dir)
@@ -58,7 +58,9 @@ VERSION_FILE = VSVersionInfo(
                     ),
                     StringStruct('OriginalFilename', 'yt-dlp%s.exe' % _x86),
                     StringStruct('ProductName', 'yt-dlp%s' % _x86),
-                    StringStruct('ProductVersion', '%s%s' % (VERSION, _x86)),
+                    StringStruct(
+                        'ProductVersion',
+                        '%s%s on Python %s' % (VERSION, _x86, platform.python_version())),
                 ])]),
         VarFileInfo([VarStruct('Translation', [0, 1200])])
     ]

setup.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from setuptools import setup, Command, find_packages
@@ -9,45 +9,44 @@ from distutils.spawn import spawn
 
 
 # Get the version from yt_dlp/version.py without importing the package
-exec(compile(open('yt_dlp/version.py').read(),
+exec(compile(open('yt_dlp/version.py').read(), 'yt_dlp/version.py', 'exec'))
-             'yt_dlp/version.py', 'exec'))
 
 
 DESCRIPTION = 'Command-line program to download videos from YouTube.com and many other other video platforms.'
 
 LONG_DESCRIPTION = '\n\n'.join((
     'Official repository: <https://github.com/yt-dlp/yt-dlp>',
-    '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
+    '**PS**: Some links in this document will not work since this is a copy of the README.md from Github',
     open('README.md', 'r', encoding='utf-8').read()))
 
 REQUIREMENTS = ['mutagen', 'pycryptodome']
 
+if sys.argv[1:2] == ['py2exe']:
+    raise NotImplementedError('py2exe is not currently supported; instead, use "pyinst.py" to build with pyinstaller')
 
-if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
+
-    print('inv')
+files_spec = [
-else:
-    files_spec = [
     ('share/bash-completion/completions', ['completions/bash/yt-dlp']),
     ('share/zsh/site-functions', ['completions/zsh/_yt-dlp']),
     ('share/fish/vendor_completions.d', ['completions/fish/yt-dlp.fish']),
     ('share/doc/yt_dlp', ['README.txt']),
     ('share/man/man1', ['yt-dlp.1'])
-    ]
+]
-    root = os.path.dirname(os.path.abspath(__file__))
+root = os.path.dirname(os.path.abspath(__file__))
-    data_files = []
+data_files = []
-    for dirname, files in files_spec:
+for dirname, files in files_spec:
     resfiles = []
     for fn in files:
         if not os.path.exists(fn):
-                warnings.warn('Skipping file %s since it is not present. Try running `make pypi-files` first.' % fn)
+            warnings.warn('Skipping file %s since it is not present. Try running `make pypi-files` first' % fn)
         else:
             resfiles.append(fn)
     data_files.append((dirname, resfiles))
 
-    params = {
+params = {
     'data_files': data_files,
-    }
+}
-    params['entry_points'] = {'console_scripts': ['yt-dlp = yt_dlp:main']}
+params['entry_points'] = {'console_scripts': ['yt-dlp = yt_dlp:main']}
 
 
 class build_lazy_extractors(Command):
@@ -61,10 +60,8 @@ class build_lazy_extractors(Command):
         pass
 
     def run(self):
-        spawn(
+        spawn([sys.executable, 'devscripts/make_lazy_extractors.py', 'yt_dlp/extractor/lazy_extractors.py'],
-            [sys.executable, 'devscripts/make_lazy_extractors.py', 'yt_dlp/extractor/lazy_extractors.py'],
+              dry_run=self.dry_run)
-            dry_run=self.dry_run,
-        )
 
 
 packages = find_packages(exclude=('youtube_dl', 'test', 'ytdlp_plugins'))
@@ -91,26 +88,16 @@ setup(
         'Development Status :: 5 - Production/Stable',
         'Environment :: Console',
         'Programming Language :: Python',
-        'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
-        'Programming Language :: Python :: 2.7',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.2',
-        'Programming Language :: Python :: 3.3',
-        'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
         'Programming Language :: Python :: Implementation',
         'Programming Language :: Python :: Implementation :: CPython',
-        'Programming Language :: Python :: Implementation :: IronPython',
-        'Programming Language :: Python :: Implementation :: Jython',
         'Programming Language :: Python :: Implementation :: PyPy',
         'License :: Public Domain',
         'Operating System :: OS Independent',
     ],
-    python_requires='>=2.6',
+    python_requires='>=3.6',
 
     cmdclass={'build_lazy_extractors': build_lazy_extractors},
     **params

supportedsites.md

@@ -82,6 +82,7 @@
  - **audiomack**
  - **audiomack:album**
  - **Audius**: Audius.co
+ - **audius:artist**: Audius.co profile/artist pages
  - **audius:playlist**: Audius.co playlists
  - **audius:track**: Audius track ID or API link. Prepend with "audius:"
  - **AWAAN**
@@ -306,6 +307,7 @@
  - **EyedoTV**
  - **facebook**
  - **FacebookPluginsVideo**
+ - **fancode:vod**
  - **faz.net**
  - **fc2**
  - **fc2:embed**
@@ -391,6 +393,7 @@
  - **HotNewHipHop**
  - **hotstar**
  - **hotstar:playlist**
+ - **hotstar:series**
  - **Howcast**
  - **HowStuffWorks**
  - **hrfernsehen**
@@ -725,6 +728,7 @@
  - **pandora.tv**: 판도라TV
  - **ParamountNetwork**
  - **parliamentlive.tv**: UK parliament videos
+ - **Parlview**
  - **Patreon**
  - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
  - **PearVideo**
@@ -748,6 +752,7 @@
  - **play.fm**
  - **player.sky.it**
  - **PlayPlusTV**
+ - **PlayStuff**
  - **PlaysTV**
  - **Playtvak**: Playtvak.cz, iDNES.cz and Lidovky.cz
  - **Playvid**
@@ -856,6 +861,7 @@
  - **safari**: safaribooksonline.com online video
  - **safari:api**
  - **safari:course**: safaribooksonline.com online courses
+ - **Saitosan**
  - **SAKTV**
  - **SaltTV**
  - **SampleFocus**
@@ -880,6 +886,7 @@
  - **Shahid**
  - **ShahidShow**
  - **Shared**: shared.sx
+ - **ShemarooMe**
  - **ShowRoomLive**
  - **simplecast**
  - **simplecast:episode**
@@ -899,6 +906,7 @@
  - **Snotr**
  - **Sohu**
  - **SonyLIV**
+ - **SonyLIVSeries**
  - **soundcloud**
  - **soundcloud:playlist**
  - **soundcloud:search**: Soundcloud search
@@ -977,6 +985,7 @@
  - **Telecinco**: telecinco.es, cuatro.com and mediaset.es
  - **Telegraaf**
  - **TeleMB**
+ - **Telemundo**
  - **TeleQuebec**
  - **TeleQuebecEmission**
  - **TeleQuebecLive**
@@ -1060,6 +1069,8 @@
  - **TVPlayHome**
  - **Tweakers**
  - **TwitCasting**
+ - **TwitCastingLive**
+ - **TwitCastingUser**
  - **twitch:clips**
  - **twitch:stream**
  - **twitch:vod**
@@ -1121,6 +1132,8 @@
  - **videomore:video**
  - **VideoPress**
  - **Vidio**
+ - **VidioLive**
+ - **VidioPremier**
  - **VidLii**
  - **vidme**
  - **vidme:user**
@@ -1160,6 +1173,7 @@
  - **VODPlatform**
  - **VoiceRepublic**
  - **Voot**
+ - **VootSeries**
  - **VoxMedia**
  - **VoxMediaVolume**
  - **vpro**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
@@ -1189,6 +1203,7 @@
  - **wdr:mobile**
  - **WDRElefant**
  - **WDRPage**
+ - **web.archive:youtube**: web.archive.org saved youtube videos
  - **Webcaster**
  - **WebcasterFeed**
  - **WebOfStories**

test/parameters.json

@@ -19,7 +19,6 @@
     "noprogress": false,
     "outtmpl": "%(id)s.%(ext)s",
     "password": null,
-    "playlistend": -1, 
     "playliststart": 1,
     "prefer_free_formats": false,
     "quiet": false,

test/test_YoutubeDL.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals
@@ -17,7 +17,7 @@ from yt_dlp.compat import compat_str, compat_urllib_error
 from yt_dlp.extractor import YoutubeIE
 from yt_dlp.extractor.common import InfoExtractor
 from yt_dlp.postprocessor.common import PostProcessor
-from yt_dlp.utils import ExtractorError, match_filter_func
+from yt_dlp.utils import ExtractorError, int_or_none, match_filter_func
 
 TEST_URL = 'http://localhost/sample.mp4'
 
@@ -29,6 +29,7 @@ class YDL(FakeYDL):
         self.msgs = []
 
     def process_info(self, info_dict):
+        info_dict.pop('__original_infodict', None)
         self.downloaded_info_dicts.append(info_dict)
 
     def to_screen(self, msg):
@@ -647,56 +648,122 @@ class TestYoutubeDL(unittest.TestCase):
         self.assertEqual(test_dict['extractor'], 'Foo')
         self.assertEqual(test_dict['playlist'], 'funny videos')
 
-    def test_prepare_filename(self):
+    outtmpl_info = {
-        info = {
         'id': '1234',
         'ext': 'mp4',
         'width': None,
         'height': 1080,
         'title1': '$PATH',
         'title2': '%PATH%',
+        'title3': 'foo/bar\\test',
         'timestamp': 1618488000,
-            'formats': [{'id': 'id1'}, {'id': 'id2'}]
+        'duration': 100000,
+        'playlist_index': 1,
+        '_last_playlist_index': 100,
+        'n_entries': 10,
+        'formats': [{'id': 'id1'}, {'id': 'id2'}, {'id': 'id3'}]
     }
 
-        def fname(templ, na_placeholder='NA'):
+    def test_prepare_outtmpl_and_filename(self):
-            params = {'outtmpl': templ}
+        def test(tmpl, expected, **params):
-            if na_placeholder != 'NA':
+            params['outtmpl'] = tmpl
-                params['outtmpl_na_placeholder'] = na_placeholder
             ydl = YoutubeDL(params)
-            return ydl.prepare_filename(info)
+            ydl._num_downloads = 1
-        self.assertEqual(fname('%(id)s.%(ext)s'), '1234.mp4')
+            self.assertEqual(ydl.validate_outtmpl(tmpl), None)
-        self.assertEqual(fname('%(id)s-%(width)s.%(ext)s'), '1234-NA.mp4')
+
-        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(id)s.%(ext)s'
+            outtmpl, tmpl_dict = ydl.prepare_outtmpl(tmpl, self.outtmpl_info)
-        # Replace missing fields with 'NA' by default
+            out = outtmpl % tmpl_dict
-        self.assertEqual(fname(NA_TEST_OUTTMPL), 'NA-NA-1234.mp4')
+            fname = ydl.prepare_filename(self.outtmpl_info)
-        # Or by provided placeholder
+
-        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder='none'), 'none-none-1234.mp4')
+            if callable(expected):
-        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder=''), '--1234.mp4')
+                self.assertTrue(expected(out))
-        self.assertEqual(fname('%(height)s.%(ext)s'), '1080.mp4')
+                self.assertTrue(expected(fname))
-        self.assertEqual(fname('%(height)d.%(ext)s'), '1080.mp4')
+            elif isinstance(expected, compat_str):
-        self.assertEqual(fname('%(height)6d.%(ext)s'), '  1080.mp4')
+                self.assertEqual((out, fname), (expected, expected))
-        self.assertEqual(fname('%(height)-6d.%(ext)s'), '1080  .mp4')
+            else:
-        self.assertEqual(fname('%(height)06d.%(ext)s'), '001080.mp4')
+                self.assertEqual((out, fname), expected)
-        self.assertEqual(fname('%(height) 06d.%(ext)s'), ' 01080.mp4')
+
-        self.assertEqual(fname('%(height)   06d.%(ext)s'), ' 01080.mp4')
+        # Auto-generated fields
-        self.assertEqual(fname('%(height)0 6d.%(ext)s'), ' 01080.mp4')
+        test('%(id)s.%(ext)s', '1234.mp4')
-        self.assertEqual(fname('%(height)0   6d.%(ext)s'), ' 01080.mp4')
+        test('%(duration_string)s', ('27:46:40', '27-46-40'))
-        self.assertEqual(fname('%(height)   0   6d.%(ext)s'), ' 01080.mp4')
+        test('%(epoch)d', int_or_none)
-        self.assertEqual(fname('%%'), '%')
+        test('%(resolution)s', '1080p')
-        self.assertEqual(fname('%%%%'), '%%')
+        test('%(playlist_index)s', '001')
-        self.assertEqual(fname('%%(height)06d.%(ext)s'), '%(height)06d.mp4')
+        test('%(autonumber)s', '00001')
-        self.assertEqual(fname('%(width)06d.%(ext)s'), 'NA.mp4')
+        test('%(autonumber+2)03d', '005', autonumber_start=3)
-        self.assertEqual(fname('%(width)06d.%%(ext)s'), 'NA.%(ext)s')
+        test('%(autonumber)s', '001', autonumber_size=3)
-        self.assertEqual(fname('%%(width)06d.%(ext)s'), '%(width)06d.mp4')
+
-        self.assertEqual(fname('Hello %(title1)s'), 'Hello $PATH')
+        # Escaping %
-        self.assertEqual(fname('Hello %(title2)s'), 'Hello %PATH%')
+        test('%%', '%')
-        self.assertEqual(fname('%(timestamp+-1000>%H-%M-%S)s'), '11-43-20')
+        test('%%%%', '%%')
-        self.assertEqual(fname('%(id+1)05d'), '01235')
+        test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
-        self.assertEqual(fname('%(width+100)05d'), 'NA')
+        test('%(width)06d.%(ext)s', 'NA.mp4')
-        self.assertEqual(fname('%(formats.0)s').replace("u", ""), "{'id' - 'id1'}")
+        test('%(width)06d.%%(ext)s', 'NA.%(ext)s')
-        self.assertEqual(fname('%(formats.-1.id)s'), 'id2')
+        test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
-        self.assertEqual(fname('%(formats.2)s'), 'NA')
+
+        # Invalid templates
+        self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%'), ValueError))
+        self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%(title)'), ValueError))
+        test('%(invalid@tmpl|def)s', 'none', outtmpl_na_placeholder='none')
+        test('%()s', 'NA')
+        test('%s', '%s')
+        test('%d', '%d')
+
+        # NA placeholder
+        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(x|def)s-%(id)s.%(ext)s'
+        test(NA_TEST_OUTTMPL, 'NA-NA-def-1234.mp4')
+        test(NA_TEST_OUTTMPL, 'none-none-def-1234.mp4', outtmpl_na_placeholder='none')
+        test(NA_TEST_OUTTMPL, '--def-1234.mp4', outtmpl_na_placeholder='')
+
+        # String formatting
+        FMT_TEST_OUTTMPL = '%%(height)%s.%%(ext)s'
+        test(FMT_TEST_OUTTMPL % 's', '1080.mp4')
+        test(FMT_TEST_OUTTMPL % 'd', '1080.mp4')
+        test(FMT_TEST_OUTTMPL % '6d', '  1080.mp4')
+        test(FMT_TEST_OUTTMPL % '-6d', '1080  .mp4')
+        test(FMT_TEST_OUTTMPL % '06d', '001080.mp4')
+        test(FMT_TEST_OUTTMPL % ' 06d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '   06d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '0 6d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '0   6d', ' 01080.mp4')
+        test(FMT_TEST_OUTTMPL % '   0   6d', ' 01080.mp4')
+
+        # Type casting
+        test('%(id)d', '1234')
+        test('%(height)c', '1')
+        test('%(ext)c', 'm')
+        test('%(id)d %(id)r', "1234 '1234'")
+        test('%(id)r %(height)r', "'1234' 1080")
+        test('%(ext)s-%(ext|def)d', 'mp4-def')
+        test('%(width|0)04d', '0000')
+        test('a%(width|)d', 'a', outtmpl_na_placeholder='none')
+
+        # Internal formatting
+        FORMATS = self.outtmpl_info['formats']
+        test('%(timestamp-1000>%H-%M-%S)s', '11-43-20')
+        test('%(id+1-height+3)05d', '00158')
+        test('%(width+100)05d', 'NA')
+        test('%(formats.0) 15s', ('% 15s' % FORMATS[0], '% 15s' % str(FORMATS[0]).replace(':', ' -')))
+        test('%(formats.0)r', (repr(FORMATS[0]), repr(FORMATS[0]).replace(':', ' -')))
+        test('%(height.0)03d', '001')
+        test('%(-height.0)04d', '-001')
+        test('%(formats.-1.id)s', FORMATS[-1]['id'])
+        test('%(formats.0.id.-1)d', FORMATS[0]['id'][-1])
+        test('%(formats.3)s', 'NA')
+        test('%(formats.:2:-1)r', repr(FORMATS[:2:-1]))
+        test('%(formats.0.id.-1+id)f', '1235.000000')
+        test('%(formats.0.id.-1+formats.1.id.-1)d', '3')
+
+        # Empty filename
+        test('%(foo|)s-%(bar|)s.%(ext)s', '-.mp4')
+        # test('%(foo|)s.%(ext)s', ('.mp4', '_.mp4'))  # fixme
+        # test('%(foo|)s', ('', '_'))  # fixme
+
+        # Path expansion and escaping
+        test('Hello %(title1)s', 'Hello $PATH')
+        test('Hello %(title2)s', 'Hello %PATH%')
+        test('%(title3)s', ('foo/bar\\test', 'foo_bar_test'))
+        test('folder/%(title3)s', ('folder/foo/bar\\test', 'folder%sfoo_bar_test' % os.path.sep))
 
     def test_format_note(self):
         ydl = YoutubeDL()
@@ -755,7 +822,7 @@ class TestYoutubeDL(unittest.TestCase):
             def process_info(self, info_dict):
                 super(YDL, self).process_info(info_dict)
 
-            def _match_entry(self, info_dict, incomplete):
+            def _match_entry(self, info_dict, incomplete=False):
                 res = super(FilterYDL, self)._match_entry(info_dict, incomplete)
                 if res is None:
                     self.downloaded_info_dicts.append(info_dict)

test/test_YoutubeDLCookieJar.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_aes.py

@@ -1,5 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
-
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_age_restriction.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_all_urls.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_cache.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_compat.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_download.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 
@@ -121,6 +121,7 @@ def generator(test_case, tname):
         params['outtmpl'] = tname + '_' + params['outtmpl']
         if is_playlist and 'playlist' not in test_case:
             params.setdefault('extract_flat', 'in_playlist')
+            params.setdefault('playlistend', test_case.get('playlist_mincount'))
             params.setdefault('skip_download', True)
 
         ydl = YoutubeDL(params, auto_init=False)

test/test_downloader_http.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_execution.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_http.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_iqiyi_sdk_interpreter.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_jsinterp.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_overwrites.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import os

test/test_post_hooks.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_postprocessors.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 
@@ -8,7 +8,14 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from yt_dlp.postprocessor import MetadataFromFieldPP, MetadataFromTitlePP
+from yt_dlp import YoutubeDL
+from yt_dlp.compat import compat_shlex_quote
+from yt_dlp.postprocessor import (
+    ExecAfterDownloadPP,
+    FFmpegThumbnailsConvertorPP,
+    MetadataFromFieldPP,
+    MetadataFromTitlePP,
+)
 
 
 class TestMetadataFromField(unittest.TestCase):
@@ -30,3 +37,35 @@ class TestMetadataFromTitle(unittest.TestCase):
     def test_format_to_regex(self):
         pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
         self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+
+
+class TestConvertThumbnail(unittest.TestCase):
+    def test_escaping(self):
+        pp = FFmpegThumbnailsConvertorPP()
+        if not pp.available:
+            print('Skipping: ffmpeg not found')
+            return
+
+        file = 'test/testdata/thumbnails/foo %d bar/foo_%d.{}'
+        tests = (('webp', 'png'), ('png', 'jpg'))
+
+        for inp, out in tests:
+            out_file = file.format(out)
+            if os.path.exists(out_file):
+                os.remove(out_file)
+            pp.convert_thumbnail(file.format(inp), out)
+            assert os.path.exists(out_file)
+
+        for _, out in tests:
+            os.remove(file.format(out))
+
+
+class TestExecAfterDownload(unittest.TestCase):
+    def test_parse_cmd(self):
+        pp = ExecAfterDownloadPP(YoutubeDL(), '')
+        info = {'filepath': 'file name'}
+        quoted_filepath = compat_shlex_quote(info['filepath'])
+
+        self.assertEqual(pp.parse_cmd('echo', info), 'echo %s' % quoted_filepath)
+        self.assertEqual(pp.parse_cmd('echo.{}', info), 'echo.%s' % quoted_filepath)
+        self.assertEqual(pp.parse_cmd('echo "%(filepath)s"', info), 'echo "%s"' % info['filepath'])

test/test_socks.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_subtitles.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_swfinterp.py.disabled

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_update.py.disabled

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_utils.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals
@@ -66,6 +66,7 @@ from yt_dlp.utils import (
     sanitize_filename,
     sanitize_path,
     sanitize_url,
+    sanitized_Request,
     expand_path,
     prepend_extension,
     replace_extension,
@@ -125,6 +126,7 @@ class TestUtil(unittest.TestCase):
         self.assertTrue(timeconvert('bougrg') is None)
 
     def test_sanitize_filename(self):
+        self.assertEqual(sanitize_filename(''), '')
         self.assertEqual(sanitize_filename('abc'), 'abc')
         self.assertEqual(sanitize_filename('abc_d-e'), 'abc_d-e')
 
@@ -238,6 +240,16 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(sanitize_url('httpss://foo.bar'), 'https://foo.bar')
         self.assertEqual(sanitize_url('rmtps://foo.bar'), 'rtmps://foo.bar')
         self.assertEqual(sanitize_url('https://foo.bar'), 'https://foo.bar')
+        self.assertEqual(sanitize_url('foo bar'), 'foo bar')
+
+    def test_extract_basic_auth(self):
+        auth_header = lambda url: sanitized_Request(url).get_header('Authorization')
+        self.assertFalse(auth_header('http://foo.bar'))
+        self.assertFalse(auth_header('http://:foo.bar'))
+        self.assertEqual(auth_header('http://@foo.bar'), 'Basic Og==')
+        self.assertEqual(auth_header('http://:pass@foo.bar'), 'Basic OnBhc3M=')
+        self.assertEqual(auth_header('http://user:@foo.bar'), 'Basic dXNlcjo=')
+        self.assertEqual(auth_header('http://user:pass@foo.bar'), 'Basic dXNlcjpwYXNz')
 
     def test_expand_path(self):
         def env(var):

test/test_verbose_output.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_write_annotations.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_youtube_lists.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_youtube_misc.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_youtube_signature.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

yt_dlp/YoutubeDL.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import absolute_import, unicode_literals
@@ -48,7 +48,6 @@ from .utils import (
     date_from_str,
     DateRange,
     DEFAULT_OUTTMPL,
-    OUTTMPL_TYPES,
     determine_ext,
     determine_protocol,
     DOT_DESKTOP_LINK_TEMPLATE,
@@ -57,26 +56,28 @@ from .utils import (
     DownloadError,
     encode_compat_str,
     encodeFilename,
-    error_to_compat_str,
     EntryNotInPlaylist,
+    error_to_compat_str,
     ExistingVideoReached,
     expand_path,
     ExtractorError,
     float_or_none,
     format_bytes,
     format_field,
-    FORMAT_RE,
+    STR_FORMAT_RE,
     formatSeconds,
     GeoRestrictedError,
     int_or_none,
     iri_to_uri,
     ISO3166Utils,
+    LazyList,
     locked_file,
     make_dir,
     make_HTTPS_handler,
     MaxDownloadsReached,
     network_exceptions,
     orderedSet,
+    OUTTMPL_TYPES,
     PagedList,
     parse_filesize,
     PerRequestProxyHandler,
@@ -84,11 +85,12 @@ from .utils import (
     PostProcessingError,
     preferredencoding,
     prepend_extension,
+    process_communicate_or_kill,
     random_uuidv4,
     register_socks_protocols,
+    RejectedVideoReached,
     render_table,
     replace_extension,
-    RejectedVideoReached,
     SameFileError,
     sanitize_filename,
     sanitize_path,
@@ -99,7 +101,7 @@ from .utils import (
     strftime_or_none,
     subtitles_filename,
     to_high_limit_path,
-    traverse_dict,
+    traverse_obj,
     UnavailableVideoError,
     url_basename,
     version_tuple,
@@ -109,7 +111,6 @@ from .utils import (
     YoutubeDLCookieProcessor,
     YoutubeDLHandler,
     YoutubeDLRedirectHandler,
-    process_communicate_or_kill,
 )
 from .cache import Cache
 from .extractor import (
@@ -177,13 +178,14 @@ class YoutubeDL(object):
     verbose:           Print additional info to stdout.
     quiet:             Do not print messages to stdout.
     no_warnings:       Do not print out anything for warnings.
-    forceurl:          Force printing final URL.
+    forceprint:        A list of templates to force print
-    forcetitle:        Force printing title.
+    forceurl:          Force printing final URL. (Deprecated)
-    forceid:           Force printing ID.
+    forcetitle:        Force printing title. (Deprecated)
-    forcethumbnail:    Force printing thumbnail URL.
+    forceid:           Force printing ID. (Deprecated)
-    forcedescription:  Force printing description.
+    forcethumbnail:    Force printing thumbnail URL. (Deprecated)
-    forcefilename:     Force printing final filename.
+    forcedescription:  Force printing description. (Deprecated)
-    forceduration:     Force printing duration.
+    forcefilename:     Force printing final filename. (Deprecated)
+    forceduration:     Force printing duration. (Deprecated)
     forcejson:         Force printing info_dict as JSON.
     dump_single_json:  Force printing the info_dict of the whole playlist
                        (or video) as a single JSON line.
@@ -387,7 +389,8 @@ class YoutubeDL(object):
                        use downloader suggested by extractor if None.
     compat_opts:       Compatibility options. See "Differences in default behavior".
                        Note that only format-sort, format-spec, no-live-chat,
-                       playlist-index, list-formats, no-youtube-channel-redirect
+                       no-attach-info-json, playlist-index, list-formats,
+                       no-direct-merge, no-youtube-channel-redirect,
                        and no-youtube-unavailable-videos works when used via the API
 
     The following parameters are not used by YoutubeDL itself, they are used by
@@ -542,8 +545,7 @@ class YoutubeDL(object):
         def preload_download_archive(fn):
             if fn is None:
                 return False
-            if self.params.get('verbose'):
+            self.write_debug('Loading archive file %r\n' % fn)
-                self._write_string('[debug] Loading archive file %r\n' % fn)
             try:
                 with locked_file(fn, 'r', encoding='utf-8') as archive_file:
                     for line in archive_file:
@@ -649,35 +651,25 @@ class YoutubeDL(object):
                       for _ in range(line_count))
         return res[:-len('\n')]
 
-    def to_screen(self, message, skip_eol=False):
-        """Print message to stdout if not in quiet mode."""
-        return self.to_stdout(
-            message, skip_eol,
-            quiet=self.params.get('quiet', False))
-
     def _write_string(self, s, out=None):
         write_string(s, out=out, encoding=self.params.get('encoding'))
 
     def to_stdout(self, message, skip_eol=False, quiet=False):
-        """Print message to stdout if not in quiet mode."""
+        """Print message to stdout"""
         if self.params.get('logger'):
             self.params['logger'].debug(message)
-        elif not quiet:
+        elif not quiet or self.params.get('verbose'):
-            message = self._bidi_workaround(message)
+            self._write_string(
-            terminator = ['\n', ''][skip_eol]
+                '%s%s' % (self._bidi_workaround(message), ('' if skip_eol else '\n')),
-            output = message + terminator
+                self._err_file if quiet else self._screen_file)
-
-            self._write_string(output, self._screen_file)
 
     def to_stderr(self, message):
-        """Print message to stderr."""
+        """Print message to stderr"""
         assert isinstance(message, compat_str)
         if self.params.get('logger'):
             self.params['logger'].error(message)
         else:
-            message = self._bidi_workaround(message)
+            self._write_string('%s\n' % self._bidi_workaround(message), self._err_file)
-            output = message + '\n'
-            self._write_string(output, self._err_file)
 
     def to_console_title(self, message):
         if not self.params.get('consoletitle', False):
@@ -739,6 +731,7 @@ class YoutubeDL(object):
                 else:
                     tb_data = traceback.format_list(traceback.extract_stack())
                     tb = ''.join(tb_data)
+            if tb:
                 self.to_stderr(tb)
         if not self.params.get('ignoreerrors', False):
             if sys.exc_info()[0] and hasattr(sys.exc_info()[1], 'exc_info') and sys.exc_info()[1].exc_info[0]:
@@ -748,6 +741,11 @@ class YoutubeDL(object):
             raise DownloadError(message, exc_info)
         self._download_retcode = 1
 
+    def to_screen(self, message, skip_eol=False):
+        """Print message to stdout if not in quiet mode"""
+        self.to_stdout(
+            message, skip_eol, quiet=self.params.get('quiet', False))
+
     def report_warning(self, message):
         '''
         Print the message to stderr, it will be prefixed with 'WARNING:'
@@ -777,6 +775,16 @@ class YoutubeDL(object):
         error_message = '%s %s' % (_msg_header, message)
         self.trouble(error_message, tb)
 
+    def write_debug(self, message):
+        '''Log debug message or Print message to stderr'''
+        if not self.params.get('verbose', False):
+            return
+        message = '[debug] %s' % message
+        if self.params.get('logger'):
+            self.params['logger'].debug(message)
+        else:
+            self._write_string('%s\n' % message)
+
     def report_file_already_downloaded(self, file_name):
         """Report file has already been fully downloaded."""
         try:
@@ -805,134 +813,139 @@ class YoutubeDL(object):
                     'Put  from __future__ import unicode_literals  at the top of your code file or consider switching to Python 3.x.')
         return outtmpl_dict
 
+    @staticmethod
+    def validate_outtmpl(tmpl):
+        ''' @return None or Exception object '''
+        try:
+            re.sub(
+                STR_FORMAT_RE.format(''),
+                lambda mobj: ('%' if not mobj.group('has_key') else '') + mobj.group(0),
+                tmpl
+            ) % collections.defaultdict(int)
+            return None
+        except ValueError as err:
+            return err
+
     def prepare_outtmpl(self, outtmpl, info_dict, sanitize=None):
         """ Make the template and info_dict suitable for substitution (outtmpl % info_dict)"""
-        template_dict = dict(info_dict)
+        info_dict = dict(info_dict)
         na = self.params.get('outtmpl_na_placeholder', 'NA')
 
-        # duration_string
+        info_dict['duration_string'] = (  # %(duration>%H-%M-%S)s is wrong if duration > 24hrs
-        template_dict['duration_string'] = (  # %(duration>%H-%M-%S)s is wrong if duration > 24hrs
+            formatSeconds(info_dict['duration'], '-' if sanitize else ':')
-            formatSeconds(info_dict['duration'], '-')
             if info_dict.get('duration', None) is not None
             else None)
-
+        info_dict['epoch'] = int(time.time())
-        # epoch
+        info_dict['autonumber'] = self.params.get('autonumber_start', 1) - 1 + self._num_downloads
-        template_dict['epoch'] = int(time.time())
+        if info_dict.get('resolution') is None:
-
+            info_dict['resolution'] = self.format_resolution(info_dict, default=None)
-        # autonumber
-        autonumber_size = self.params.get('autonumber_size')
-        if autonumber_size is None:
-            autonumber_size = 5
-        template_dict['autonumber'] = self.params.get('autonumber_start', 1) - 1 + self._num_downloads
-
-        # resolution if not defined
-        if template_dict.get('resolution') is None:
-            if template_dict.get('width') and template_dict.get('height'):
-                template_dict['resolution'] = '%dx%d' % (template_dict['width'], template_dict['height'])
-            elif template_dict.get('height'):
-                template_dict['resolution'] = '%sp' % template_dict['height']
-            elif template_dict.get('width'):
-                template_dict['resolution'] = '%dx?' % template_dict['width']
 
         # For fields playlist_index and autonumber convert all occurrences
         # of %(field)s to %(field)0Nd for backward compatibility
         field_size_compat_map = {
-            'playlist_index': len(str(template_dict.get('_last_playlist_index') or '')),
+            'playlist_index': len(str(info_dict.get('_last_playlist_index') or '')),
-            'autonumber': autonumber_size,
+            'autonumber': self.params.get('autonumber_size') or 5,
         }
-        FIELD_SIZE_COMPAT_RE = r'(?<!%)%\((?P<field>autonumber|playlist_index)\)s'
-        mobj = re.search(FIELD_SIZE_COMPAT_RE, outtmpl)
-        if mobj:
-            outtmpl = re.sub(
-                FIELD_SIZE_COMPAT_RE,
-                r'%%(\1)0%dd' % field_size_compat_map[mobj.group('field')],
-                outtmpl)
 
-        numeric_fields = list(self._NUMERIC_FIELDS)
+        TMPL_DICT = {}
-        if sanitize is None:
+        EXTERNAL_FORMAT_RE = re.compile(STR_FORMAT_RE.format('[^)]*'))
-            sanitize = lambda k, v: v
-
-        EXTERNAL_FORMAT_RE = FORMAT_RE.format('(?P<key>[^)]*)')
-        # Field is of the form key1.key2...
-        # where keys (except first) can be string, int or slice
-        FIELD_RE = r'\w+(?:\.(?:\w+|[-\d]*(?::[-\d]*){0,2}))*'
-        INTERNAL_FORMAT_RE = re.compile(r'''(?x)
-            (?P<negate>-)?
-            (?P<fields>{0})
-            (?P<maths>(?:[-+]-?(?:\d+(?:\.\d+)?|{0}))*)
-            (?:>(?P<strf_format>.+?))?
-            (?:\|(?P<default>.*?))?
-            $'''.format(FIELD_RE))
-        MATH_OPERATORS_RE = re.compile(r'(?<![-+])([-+])')
         MATH_FUNCTIONS = {
             '+': float.__add__,
             '-': float.__sub__,
         }
-        for outer_mobj in re.finditer(EXTERNAL_FORMAT_RE, outtmpl):
+        # Field is of the form key1.key2...
-            final_key = outer_mobj.group('key')
+        # where keys (except first) can be string, int or slice
-            str_type = outer_mobj.group('type')
+        FIELD_RE = r'\w+(?:\.(?:\w+|{num}|{num}?(?::{num}?){{1,2}}))*'.format(num=r'(?:-?\d+)')
-            value = None
+        MATH_FIELD_RE = r'''{field}|{num}'''.format(field=FIELD_RE, num=r'-?\d+(?:.\d+)?')
-            mobj = re.match(INTERNAL_FORMAT_RE, final_key)
+        MATH_OPERATORS_RE = r'(?:%s)' % '|'.join(map(re.escape, MATH_FUNCTIONS.keys()))
-            if mobj is not None:
+        INTERNAL_FORMAT_RE = re.compile(r'''(?x)
-                mobj = mobj.groupdict()
+            (?P<negate>-)?
+            (?P<fields>{field})
+            (?P<maths>(?:{math_op}{math_field})*)
+            (?:>(?P<strf_format>.+?))?
+            (?:\|(?P<default>.*?))?
+            $'''.format(field=FIELD_RE, math_op=MATH_OPERATORS_RE, math_field=MATH_FIELD_RE))
+
+        get_key = lambda k: traverse_obj(
+            info_dict, k.split('.'), is_user_input=True, traverse_string=True)
+
+        def get_value(mdict):
             # Object traversal
-                fields = mobj['fields'].split('.')
+            value = get_key(mdict['fields'])
-                value = traverse_dict(template_dict, fields)
             # Negative
-                if mobj['negate']:
+            if mdict['negate']:
                 value = float_or_none(value)
                 if value is not None:
                     value *= -1
             # Do maths
-                if mobj['maths']:
+            offset_key = mdict['maths']
+            if offset_key:
                 value = float_or_none(value)
                 operator = None
-                    for item in MATH_OPERATORS_RE.split(mobj['maths'])[1:]:
+                while offset_key:
-                        if item == '':
+                    item = re.match(
-                            value = None
+                        MATH_FIELD_RE if operator else MATH_OPERATORS_RE,
-                        if value is None:
+                        offset_key).group(0)
-                            break
+                    offset_key = offset_key[len(item):]
-                        if operator:
+                    if operator is None:
+                        operator = MATH_FUNCTIONS[item]
+                        continue
                     item, multiplier = (item[1:], -1) if item[0] == '-' else (item, 1)
                     offset = float_or_none(item)
                     if offset is None:
-                                offset = float_or_none(traverse_dict(template_dict, item.split('.')))
+                        offset = float_or_none(get_key(item))
                     try:
                         value = operator(value, multiplier * offset)
                     except (TypeError, ZeroDivisionError):
-                                value = None
+                        return None
                     operator = None
-                        else:
-                            operator = MATH_FUNCTIONS[item]
             # Datetime formatting
-                if mobj['strf_format']:
+            if mdict['strf_format']:
-                    value = strftime_or_none(value, mobj['strf_format'])
+                value = strftime_or_none(value, mdict['strf_format'])
-                # Set default
+
-                if value is None and mobj['default'] is not None:
+            return value
-                    value = mobj['default']
+
-            # Sanitize
+        def create_key(outer_mobj):
-            if str_type in 'crs' and value is not None:  # string
+            if not outer_mobj.group('has_key'):
-                value = sanitize('%{}'.format(str_type) % fields[-1], value)
+                return '%{}'.format(outer_mobj.group(0))
-            else:  # numeric
+
-                numeric_fields.append(final_key)
+            key = outer_mobj.group('key')
+            fmt = outer_mobj.group('format')
+            mobj = re.match(INTERNAL_FORMAT_RE, key)
+            if mobj is None:
+                value, default = None, na
+            else:
+                mobj = mobj.groupdict()
+                default = mobj['default'] if mobj['default'] is not None else na
+                value = get_value(mobj)
+
+            if fmt == 's' and value is not None and key in field_size_compat_map.keys():
+                fmt = '0{:d}d'.format(field_size_compat_map[key])
+
+            value = default if value is None else value
+            key += '\0%s' % fmt
+
+            if fmt == 'c':
+                value = compat_str(value)
+                if value is None:
+                    value, fmt = default, 's'
+                else:
+                    value = value[0]
+            elif fmt[-1] not in 'rs':  # numeric
                 value = float_or_none(value)
-            if value is not None:
+                if value is None:
-                template_dict[final_key] = value
+                    value, fmt = default, 's'
+            if sanitize:
+                if fmt[-1] == 'r':
+                    # If value is an object, sanitize might convert it to a string
+                    # So we convert it to repr first
+                    value, fmt = repr(value), '%ss' % fmt[:-1]
+                if fmt[-1] in 'csr':
+                    value = sanitize(key, value)
+            TMPL_DICT[key] = value
+            return '%({key}){fmt}'.format(key=key, fmt=fmt)
 
-        # Missing numeric fields used together with integer presentation types
+        return EXTERNAL_FORMAT_RE.sub(create_key, outtmpl), TMPL_DICT
-        # in format specification will break the argument substitution since
-        # string NA placeholder is returned for missing fields. We will patch
-        # output template for missing fields to meet string presentation type.
-        for numeric_field in numeric_fields:
-            if template_dict.get(numeric_field) is None:
-                outtmpl = re.sub(
-                    FORMAT_RE.format(re.escape(numeric_field)),
-                    r'%({0})s'.format(numeric_field), outtmpl)
-
-        template_dict = collections.defaultdict(lambda: na, (
-            (k, v if isinstance(v, compat_numeric_types) else sanitize(k, v))
-            for k, v in template_dict.items() if v is not None))
-        return outtmpl, template_dict
 
     def _prepare_filename(self, info_dict, tmpl_type='default'):
         try:
@@ -958,7 +971,7 @@ class YoutubeDL(object):
 
             force_ext = OUTTMPL_TYPES.get(tmpl_type)
             if force_ext is not None:
-                filename = replace_extension(filename, force_ext, template_dict.get('ext'))
+                filename = replace_extension(filename, force_ext, info_dict.get('ext'))
 
             # https://github.com/blackjack4494/youtube-dlc/issues/85
             trim_file_name = self.params.get('trim_file_name', False)
@@ -1005,11 +1018,12 @@ class YoutubeDL(object):
             path = encodeFilename(path, True).decode(preferredencoding())
         return sanitize_path(path, force=self.params.get('windowsfilenames'))
 
-    def _match_entry(self, info_dict, incomplete):
+    def _match_entry(self, info_dict, incomplete=False, silent=False):
         """ Returns None if the file should be downloaded """
 
-        def check_filter():
         video_title = info_dict.get('title', info_dict.get('id', 'video'))
+
+        def check_filter():
             if 'title' in info_dict:
                 # This can happen when we're just evaluating the playlist
                 title = info_dict['title']
@@ -1036,8 +1050,6 @@ class YoutubeDL(object):
                     return 'Skipping %s, because it has exceeded the maximum view count (%d/%d)' % (video_title, view_count, max_views)
             if age_restricted(info_dict.get('age_limit'), self.params.get('age_limit')):
                 return 'Skipping "%s" because it is age restricted' % video_title
-            if self.in_download_archive(info_dict):
-                return '%s has already been recorded in archive' % video_title
 
             if not incomplete:
                 match_filter = self.params.get('match_filter')
@@ -1047,13 +1059,17 @@ class YoutubeDL(object):
                         return ret
             return None
 
+        if self.in_download_archive(info_dict):
+            reason = '%s has already been recorded in the archive' % video_title
+            break_opt, break_err = 'break_on_existing', ExistingVideoReached
+        else:
             reason = check_filter()
+            break_opt, break_err = 'break_on_reject', RejectedVideoReached
         if reason is not None:
+            if not silent:
                 self.to_screen('[download] ' + reason)
-            if reason.endswith('has already been recorded in the archive') and self.params.get('break_on_existing', False):
+            if self.params.get(break_opt, False):
-                raise ExistingVideoReached()
+                raise break_err()
-            elif self.params.get('break_on_reject', False):
-                raise RejectedVideoReached()
         return reason
 
     @staticmethod
@@ -1154,6 +1170,7 @@ class YoutubeDL(object):
         self.add_extra_info(ie_result, {
             'extractor': ie.IE_NAME,
             'webpage_url': url,
+            'original_url': url,
             'webpage_url_basename': url_basename(url),
             'extractor_key': ie.ie_key(),
         })
@@ -1173,17 +1190,36 @@ class YoutubeDL(object):
             extract_flat = self.params.get('extract_flat', False)
             if ((extract_flat == 'in_playlist' and 'playlist' in extra_info)
                     or extract_flat is True):
-                self.__forced_printings(ie_result, self.prepare_filename(ie_result), incomplete=True)
+                info_copy = ie_result.copy()
+                self.add_extra_info(info_copy, extra_info)
+                self.add_default_extra_info(
+                    info_copy, self.get_info_extractor(ie_result.get('ie_key')), ie_result['url'])
+                self.__forced_printings(info_copy, self.prepare_filename(info_copy), incomplete=True)
                 return ie_result
 
         if result_type == 'video':
             self.add_extra_info(ie_result, extra_info)
-            return self.process_video_result(ie_result, download=download)
+            ie_result = self.process_video_result(ie_result, download=download)
+            additional_urls = (ie_result or {}).get('additional_urls')
+            if additional_urls:
+                # TODO: Improve MetadataFromFieldPP to allow setting a list
+                if isinstance(additional_urls, compat_str):
+                    additional_urls = [additional_urls]
+                self.to_screen(
+                    '[info] %s: %d additional URL(s) requested' % (ie_result['id'], len(additional_urls)))
+                self.write_debug('Additional URLs: "%s"' % '", "'.join(additional_urls))
+                ie_result['additional_entries'] = [
+                    self.extract_info(
+                        url, download, extra_info,
+                        force_generic_extractor=self.params.get('force_generic_extractor'))
+                    for url in additional_urls
+                ]
+            return ie_result
         elif result_type == 'url':
             # We have to add extra_info to the results because it may be
             # contained in a playlist
-            return self.extract_info(ie_result['url'],
+            return self.extract_info(
-                                     download,
+                ie_result['url'], download,
                 ie_key=ie_result.get('ie_key'),
                 extra_info=extra_info)
         elif result_type == 'url_transparent':
@@ -1229,6 +1265,7 @@ class YoutubeDL(object):
 
             self._playlist_level += 1
             self._playlist_urls.add(webpage_url)
+            self._sanitize_thumbnails(ie_result)
             try:
                 return self.__process_playlist(ie_result, download)
             finally:
@@ -1280,7 +1317,7 @@ class YoutubeDL(object):
 
         playlist_results = []
 
-        playliststart = self.params.get('playliststart', 1) - 1
+        playliststart = self.params.get('playliststart', 1)
         playlistend = self.params.get('playlistend')
         # For backwards compatibility, interpret -1 as whole list
         if playlistend == -1:
@@ -1300,50 +1337,43 @@ class YoutubeDL(object):
             playlistitems = orderedSet(iter_playlistitems(playlistitems_str))
 
         ie_entries = ie_result['entries']
+        msg = (
+            'Downloading %d videos' if not isinstance(ie_entries, list)
+            else 'Collected %d videos; downloading %%d of them' % len(ie_entries))
+        if not isinstance(ie_entries, (list, PagedList)):
+            ie_entries = LazyList(ie_entries)
 
-        def make_playlistitems_entries(list_ie_entries):
-            num_entries = len(list_ie_entries)
-            for i in playlistitems:
-                if -num_entries < i <= num_entries:
-                    yield list_ie_entries[i - 1]
-                elif incomplete_entries:
-                    raise EntryNotInPlaylist()
-
-        if isinstance(ie_entries, list):
-            n_all_entries = len(ie_entries)
-            if playlistitems:
-                entries = list(make_playlistitems_entries(ie_entries))
-            else:
-                entries = ie_entries[playliststart:playlistend]
-            n_entries = len(entries)
-            msg = 'Collected %d videos; downloading %d of them' % (n_all_entries, n_entries)
-        elif isinstance(ie_entries, PagedList):
-            if playlistitems:
         entries = []
-                for item in playlistitems:
+        for i in playlistitems or itertools.count(playliststart):
-                    entries.extend(ie_entries.getslice(
+            if playlistitems is None and playlistend is not None and playlistend < i:
-                        item - 1, item
+                break
-                    ))
+            entry = None
-            else:
+            try:
-                entries = ie_entries.getslice(
+                entry = ie_entries[i - 1]
-                    playliststart, playlistend)
+                if entry is None:
-            n_entries = len(entries)
-            msg = 'Downloading %d videos' % n_entries
-        else:  # iterable
-            if playlistitems:
-                entries = list(make_playlistitems_entries(list(itertools.islice(
-                    ie_entries, 0, max(playlistitems)))))
-            else:
-                entries = list(itertools.islice(
-                    ie_entries, playliststart, playlistend))
-            n_entries = len(entries)
-            msg = 'Downloading %d videos' % n_entries
-
-        if any((entry is None for entry in entries)):
                     raise EntryNotInPlaylist()
-        if not playlistitems and (playliststart or playlistend):
+            except (IndexError, EntryNotInPlaylist):
-            playlistitems = list(range(1 + playliststart, 1 + playliststart + len(entries)))
+                if incomplete_entries:
+                    raise EntryNotInPlaylist()
+                elif not playlistitems:
+                    break
+            entries.append(entry)
+            try:
+                if entry is not None:
+                    self._match_entry(entry, incomplete=True, silent=True)
+            except (ExistingVideoReached, RejectedVideoReached):
+                break
         ie_result['entries'] = entries
+
+        # Save playlist_index before re-ordering
+        entries = [
+            ((playlistitems[i - 1] if playlistitems else i), entry)
+            for i, entry in enumerate(entries, 1)
+            if entry is not None]
+        n_entries = len(entries)
+
+        if not playlistitems and (playliststart or playlistend):
+            playlistitems = list(range(playliststart, playliststart + n_entries))
         ie_result['requested_entries'] = playlistitems
 
         if self.params.get('allow_playlist_files', True):
@@ -1370,6 +1400,9 @@ class YoutubeDL(object):
                     except (OSError, IOError):
                         self.report_error('Cannot write playlist metadata to JSON file ' + infofn)
 
+            # TODO: This should be passed to ThumbnailsConvertor if necessary
+            self._write_thumbnails(ie_copy, self.prepare_filename(ie_copy, 'pl_thumbnail'))
+
             if self.params.get('writedescription', False):
                 descfn = self.prepare_filename(ie_copy, 'pl_description')
                 if not self._ensure_dir_exists(encodeFilename(descfn)):
@@ -1387,11 +1420,6 @@ class YoutubeDL(object):
                         self.report_error('Cannot write playlist description file ' + descfn)
                         return
 
-        # Save playlist_index before re-ordering
-        entries = [
-            ((playlistitems[i - 1] if playlistitems else i), entry)
-            for i, entry in enumerate(entries, 1)]
-
         if self.params.get('playlistreverse', False):
             entries = entries[::-1]
         if self.params.get('playlistrandom', False):
@@ -1399,7 +1427,7 @@ class YoutubeDL(object):
 
         x_forwarded_for = ie_result.get('__x_forwarded_for_ip')
 
-        self.to_screen('[%s] playlist %s: %s' % (ie_result['extractor'], playlist, msg))
+        self.to_screen('[%s] playlist %s: %s' % (ie_result['extractor'], playlist, msg % n_entries))
         failures = 0
         max_failures = self.params.get('skip_playlist_after_errors') or float('inf')
         for i, entry_tuple in enumerate(entries, 1):
@@ -1717,7 +1745,11 @@ class YoutubeDL(object):
                     expand_path(paths.get('home', '').strip()),
                     expand_path(paths.get('temp', '').strip()),
                     'ytdl.%s.f%s.check-format' % (random_uuidv4(), f['format_id']))
+                try:
                     dl, _ = self.dl(temp_file, f, test=True)
+                except (ExtractorError, IOError, OSError, ValueError) + network_exceptions:
+                    dl = False
+                finally:
                     if os.path.exists(temp_file):
                         os.remove(temp_file)
                 if dl:
@@ -1882,6 +1914,27 @@ class YoutubeDL(object):
         self.cookiejar.add_cookie_header(pr)
         return pr.get_header('Cookie')
 
+    @staticmethod
+    def _sanitize_thumbnails(info_dict):
+        thumbnails = info_dict.get('thumbnails')
+        if thumbnails is None:
+            thumbnail = info_dict.get('thumbnail')
+            if thumbnail:
+                info_dict['thumbnails'] = thumbnails = [{'url': thumbnail}]
+        if thumbnails:
+            thumbnails.sort(key=lambda t: (
+                t.get('preference') if t.get('preference') is not None else -1,
+                t.get('width') if t.get('width') is not None else -1,
+                t.get('height') if t.get('height') is not None else -1,
+                t.get('id') if t.get('id') is not None else '',
+                t.get('url')))
+            for i, t in enumerate(thumbnails):
+                t['url'] = sanitize_url(t['url'])
+                if t.get('width') and t.get('height'):
+                    t['resolution'] = '%dx%d' % (t['width'], t['height'])
+                if t.get('id') is None:
+                    t['id'] = '%d' % i
+
     def process_video_result(self, info_dict, download=True):
         assert info_dict.get('_type', 'video') == 'video'
 
@@ -1918,29 +1971,14 @@ class YoutubeDL(object):
             info_dict['playlist'] = None
             info_dict['playlist_index'] = None
 
-        thumbnails = info_dict.get('thumbnails')
+        self._sanitize_thumbnails(info_dict)
-        if thumbnails is None:
-            thumbnail = info_dict.get('thumbnail')
-            if thumbnail:
-                info_dict['thumbnails'] = thumbnails = [{'url': thumbnail}]
-        if thumbnails:
-            thumbnails.sort(key=lambda t: (
-                t.get('preference') if t.get('preference') is not None else -1,
-                t.get('width') if t.get('width') is not None else -1,
-                t.get('height') if t.get('height') is not None else -1,
-                t.get('id') if t.get('id') is not None else '', t.get('url')))
-            for i, t in enumerate(thumbnails):
-                t['url'] = sanitize_url(t['url'])
-                if t.get('width') and t.get('height'):
-                    t['resolution'] = '%dx%d' % (t['width'], t['height'])
-                if t.get('id') is None:
-                    t['id'] = '%d' % i
 
         if self.params.get('list_thumbnails'):
             self.list_thumbnails(info_dict)
             return
 
         thumbnail = info_dict.get('thumbnail')
+        thumbnails = info_dict.get('thumbnails')
         if thumbnail:
             info_dict['thumbnail'] = sanitize_url(thumbnail)
         elif thumbnails:
@@ -2072,6 +2110,9 @@ class YoutubeDL(object):
             # element in the 'formats' field in info_dict is info_dict itself,
             # which can't be exported to json
             info_dict['formats'] = formats
+
+        info_dict, _ = self.pre_process(info_dict)
+
         if self.params.get('listformats'):
             if not info_dict.get('formats'):
                 raise ExtractorError('No video formats found', expected=True)
@@ -2081,8 +2122,7 @@ class YoutubeDL(object):
         req_format = self.params.get('format')
         if req_format is None:
             req_format = self._default_format_spec(info_dict, download=download)
-            if self.params.get('verbose'):
+            self.write_debug('Default format spec: %s' % req_format)
-                self.to_screen('[debug] Default format spec: %s' % req_format)
 
         format_selector = self.build_format_selector(req_format)
 
@@ -2120,14 +2160,13 @@ class YoutubeDL(object):
                 self.report_warning('Requested format is not available')
         elif download:
             self.to_screen(
-                '[info] %s: Downloading format(s) %s'
+                '[info] %s: Downloading %d format(s): %s' % (
-                % (info_dict['id'], ", ".join([f['format_id'] for f in formats_to_download])))
+                    info_dict['id'], len(formats_to_download),
-            if len(formats_to_download) > 1:
+                    ", ".join([f['format_id'] for f in formats_to_download])))
-                self.to_screen(
-                    '[info] %s: Downloading video in %s formats'
-                    % (info_dict['id'], len(formats_to_download)))
             for fmt in formats_to_download:
                 new_info = dict(info_dict)
+                # Save a reference to the original info_dict so that it can be modified in process_info if needed
+                new_info['__original_infodict'] = info_dict
                 new_info.update(fmt)
                 self.process_info(new_info)
         # We update the info dict with the best quality format (backwards compatibility)
@@ -2172,6 +2211,7 @@ class YoutubeDL(object):
             requested_langs = ['en']
         else:
             requested_langs = [list(all_sub_langs)[0]]
+        self.write_debug('Downloading subtitles: %s' % ', '.join(requested_langs))
 
         formats_query = self.params.get('subtitlesformat', 'best')
         formats_preference = formats_query.split('/') if formats_query else []
@@ -2198,32 +2238,43 @@ class YoutubeDL(object):
         return subs
 
     def __forced_printings(self, info_dict, filename, incomplete):
-        def print_mandatory(field):
+        def print_mandatory(field, actual_field=None):
+            if actual_field is None:
+                actual_field = field
             if (self.params.get('force%s' % field, False)
-                    and (not incomplete or info_dict.get(field) is not None)):
+                    and (not incomplete or info_dict.get(actual_field) is not None)):
-                self.to_stdout(info_dict[field])
+                self.to_stdout(info_dict[actual_field])
 
         def print_optional(field):
             if (self.params.get('force%s' % field, False)
                     and info_dict.get(field) is not None):
                 self.to_stdout(info_dict[field])
 
+        info_dict = info_dict.copy()
+        if filename is not None:
+            info_dict['filename'] = filename
+        if info_dict.get('requested_formats') is not None:
+            # For RTMP URLs, also include the playpath
+            info_dict['urls'] = '\n'.join(f['url'] + f.get('play_path', '') for f in info_dict['requested_formats'])
+        elif 'url' in info_dict:
+            info_dict['urls'] = info_dict['url'] + info_dict.get('play_path', '')
+
+        for tmpl in self.params.get('forceprint', []):
+            if re.match(r'\w+$', tmpl):
+                tmpl = '%({})s'.format(tmpl)
+            tmpl, info_copy = self.prepare_outtmpl(tmpl, info_dict)
+            self.to_stdout(tmpl % info_copy)
+
         print_mandatory('title')
         print_mandatory('id')
-        if self.params.get('forceurl', False) and not incomplete:
+        print_mandatory('url', 'urls')
-            if info_dict.get('requested_formats') is not None:
-                for f in info_dict['requested_formats']:
-                    self.to_stdout(f['url'] + f.get('play_path', ''))
-            else:
-                # For RTMP URLs, also include the playpath
-                self.to_stdout(info_dict['url'] + info_dict.get('play_path', ''))
         print_optional('thumbnail')
         print_optional('description')
-        if self.params.get('forcefilename', False) and filename is not None:
+        print_optional('filename')
-            self.to_stdout(filename)
         if self.params.get('forceduration', False) and info_dict.get('duration') is not None:
             self.to_stdout(formatSeconds(info_dict['duration']))
         print_mandatory('format')
+
         if self.params.get('forcejson', False):
             self.post_extract(info_dict)
             self.to_stdout(json.dumps(info_dict, default=repr))
@@ -2249,8 +2300,8 @@ class YoutubeDL(object):
         if not test:
             for ph in self._progress_hooks:
                 fd.add_progress_hook(ph)
-            if self.params.get('verbose'):
+            urls = '", "'.join([f['url'] for f in info.get('requested_formats', [])] or [info['url']])
-                self.to_screen('[debug] Invoking downloader on %r' % info.get('url'))
+            self.write_debug('Invoking downloader on "%s"' % urls)
         new_info = dict(info)
         if new_info.get('http_headers') is None:
             new_info['http_headers'] = self._calc_headers(new_info)
@@ -2274,14 +2325,12 @@ class YoutubeDL(object):
         if 'format' not in info_dict:
             info_dict['format'] = info_dict['ext']
 
-        if self._match_entry(info_dict, incomplete=False) is not None:
+        if self._match_entry(info_dict) is not None:
             return
 
         self.post_extract(info_dict)
         self._num_downloads += 1
 
-        info_dict, _ = self.pre_process(info_dict)
-
         # info_dict['_filename'] needs to be set for backward compatibility
         info_dict['_filename'] = full_filename = self.prepare_filename(info_dict, warn=True)
         temp_filename = self.prepare_filename(info_dict, 'temp')
@@ -2376,7 +2425,7 @@ class YoutubeDL(object):
                             self.dl(sub_filename, sub_info.copy(), subtitle=True)
                             sub_info['filepath'] = sub_filename
                             files_to_move[sub_filename] = sub_filename_final
-                        except tuple([ExtractorError, IOError, OSError, ValueError] + network_exceptions) as err:
+                        except (ExtractorError, IOError, OSError, ValueError) + network_exceptions as err:
                             self.report_warning('Unable to download subtitle for "%s": %s' %
                                                 (sub_lang, error_to_compat_str(err)))
                             continue
@@ -2491,17 +2540,6 @@ class YoutubeDL(object):
 
                 success = True
                 if info_dict.get('requested_formats') is not None:
-                    downloaded = []
-                    merger = FFmpegMergerPP(self)
-                    if self.params.get('allow_unplayable_formats'):
-                        self.report_warning(
-                            'You have requested merging of multiple formats '
-                            'while also allowing unplayable formats to be downloaded. '
-                            'The formats won\'t be merged to prevent data corruption.')
-                    elif not merger.available:
-                        self.report_warning(
-                            'You have requested merging of multiple formats but ffmpeg is not installed. '
-                            'The formats won\'t be merged.')
 
                     def compatible_formats(formats):
                         # TODO: some formats actually allow this (mkv, webm, ogg, mp4), but not all of them.
@@ -2549,9 +2587,39 @@ class YoutubeDL(object):
                     temp_filename = correct_ext(temp_filename)
                     dl_filename = existing_file(full_filename, temp_filename)
                     info_dict['__real_download'] = False
+
+                    _protocols = set(determine_protocol(f) for f in requested_formats)
+                    if len(_protocols) == 1:
+                        info_dict['protocol'] = _protocols.pop()
+                    directly_mergable = (
+                        'no-direct-merge' not in self.params.get('compat_opts', [])
+                        and info_dict.get('protocol') is not None  # All requested formats have same protocol
+                        and not self.params.get('allow_unplayable_formats')
+                        and get_suitable_downloader(info_dict, self.params).__name__ == 'FFmpegFD')
+                    if directly_mergable:
+                        info_dict['url'] = requested_formats[0]['url']
+                        # Treat it as a single download
+                        dl_filename = existing_file(full_filename, temp_filename)
+                        if dl_filename is None:
+                            success, real_download = self.dl(temp_filename, info_dict)
+                            info_dict['__real_download'] = real_download
+                    else:
+                        downloaded = []
+                        merger = FFmpegMergerPP(self)
+                        if self.params.get('allow_unplayable_formats'):
+                            self.report_warning(
+                                'You have requested merging of multiple formats '
+                                'while also allowing unplayable formats to be downloaded. '
+                                'The formats won\'t be merged to prevent data corruption.')
+                        elif not merger.available:
+                            self.report_warning(
+                                'You have requested merging of multiple formats but ffmpeg is not installed. '
+                                'The formats won\'t be merged.')
+
                         if dl_filename is None:
                             for f in requested_formats:
                                 new_info = dict(info_dict)
+                                del new_info['requested_formats']
                                 new_info.update(f)
                                 fname = prepend_extension(
                                     self.prepare_filename(new_info, 'temp'),
@@ -2719,18 +2787,20 @@ class YoutubeDL(object):
 
     @staticmethod
     def filter_requested_info(info_dict, actually_filter=True):
-        if not actually_filter:
+        remove_keys = ['__original_infodict']  # Always remove this since this may contain a copy of the entire dict
+        keep_keys = ['_type'],  # Always keep this to facilitate load-info-json
+        if actually_filter:
+            remove_keys += ('requested_formats', 'requested_subtitles', 'requested_entries', 'filepath', 'entries', 'original_url')
+            empty_values = (None, {}, [], set(), tuple())
+            reject = lambda k, v: k not in keep_keys and (
+                k.startswith('_') or k in remove_keys or v in empty_values)
+        else:
             info_dict['epoch'] = int(time.time())
-            return info_dict
+            reject = lambda k, v: k in remove_keys
-        exceptions = {
-            'remove': ['requested_formats', 'requested_subtitles', 'requested_entries', 'filepath', 'entries'],
-            'keep': ['_type'],
-        }
-        keep_key = lambda k: k in exceptions['keep'] or not (k.startswith('_') or k in exceptions['remove'])
         filter_fn = lambda obj: (
-            list(map(filter_fn, obj)) if isinstance(obj, (list, tuple))
+            list(map(filter_fn, obj)) if isinstance(obj, (list, tuple, set))
             else obj if not isinstance(obj, dict)
-            else dict((k, filter_fn(v)) for k, v in obj.items() if keep_key(k)))
+            else dict((k, filter_fn(v)) for k, v in obj.items() if not reject(k, v)))
         return filter_fn(info_dict)
 
     def run_pp(self, pp, infodict):
@@ -2763,13 +2833,14 @@ class YoutubeDL(object):
                     actual_post_extract(video_dict or {})
                 return
 
-            if '__post_extractor' not in info_dict:
+            post_extractor = info_dict.get('__post_extractor') or (lambda: {})
-                return
+            extra = post_extractor().items()
-            post_extractor = info_dict['__post_extractor']
+            info_dict.update(extra)
-            if post_extractor:
+            info_dict.pop('__post_extractor', None)
-                info_dict.update(post_extractor().items())
+
-            del info_dict['__post_extractor']
+            original_infodict = info_dict.get('__original_infodict') or {}
-            return
+            original_infodict.update(extra)
+            original_infodict.pop('__post_extractor', None)
 
         actual_post_extract(info_dict or {})
 
@@ -2982,10 +3053,17 @@ class YoutubeDL(object):
             return
         self.to_screen(
             'Available %s for %s:' % (name, video_id))
+
+        def _row(lang, formats):
+            exts, names = zip(*((f['ext'], f.get('name', 'unknown')) for f in reversed(formats)))
+            if len(set(names)) == 1:
+                names = [] if names[0] == 'unknown' else names[:1]
+            return [lang, ', '.join(names), ', '.join(exts)]
+
         self.to_screen(render_table(
-            ['Language', 'formats'],
+            ['Language', 'Name', 'Formats'],
-            [[lang, ', '.join(f['ext'] for f in reversed(formats))]
+            [_row(lang, formats) for lang, formats in subtitles.items()],
-                for lang, formats in subtitles.items()]))
+            hideEmpty=True))
 
     def urlopen(self, req):
         """ Start an HTTP download """
@@ -3164,7 +3242,7 @@ class YoutubeDL(object):
             thumb_ext = determine_ext(t['url'], 'jpg')
             suffix = '%s.' % t['id'] if multiple else ''
             thumb_display_id = '%s ' % t['id'] if multiple else ''
-            t['filepath'] = thumb_filename = replace_extension(filename, suffix + thumb_ext, info_dict.get('ext'))
+            thumb_filename = replace_extension(filename, suffix + thumb_ext, info_dict.get('ext'))
 
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
                 ret.append(suffix + thumb_ext)
@@ -3180,6 +3258,7 @@ class YoutubeDL(object):
                     ret.append(suffix + thumb_ext)
                     self.to_screen('[%s] %s: Writing thumbnail %sto: %s' %
                                    (info_dict['extractor'], info_dict['id'], thumb_display_id, thumb_filename))
+                    t['filepath'] = thumb_filename
                 except network_exceptions as err:
                     self.report_warning('Unable to download thumbnail "%s": %s' %
                                         (t['url'], error_to_compat_str(err)))

yt_dlp/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals
@@ -24,6 +24,7 @@ from .utils import (
     DateRange,
     decodeOption,
     DownloadError,
+    error_to_compat_str,
     ExistingVideoReached,
     expand_path,
     match_filter_func,
@@ -31,20 +32,26 @@ from .utils import (
     preferredencoding,
     read_batch_urls,
     RejectedVideoReached,
-    REMUX_EXTENSIONS,
     render_table,
     SameFileError,
     setproctitle,
     std_headers,
     write_string,
 )
-from .update import update_self
+from .update import run_update
 from .downloader import (
     FileDownloader,
 )
 from .extractor import gen_extractors, list_extractors
 from .extractor.common import InfoExtractor
 from .extractor.adobepass import MSO_INFO
+from .postprocessor.ffmpeg import (
+    FFmpegExtractAudioPP,
+    FFmpegSubtitlesConvertorPP,
+    FFmpegThumbnailsConvertorPP,
+    FFmpegVideoConvertorPP,
+    FFmpegVideoRemuxerPP,
+)
 from .postprocessor.metadatafromfield import MetadataFromFieldPP
 from .YoutubeDL import YoutubeDL
 
@@ -209,25 +216,25 @@ def _real_main(argv=None):
     if opts.playlistend not in (-1, None) and opts.playlistend < opts.playliststart:
         raise ValueError('Playlist end must be greater than playlist start')
     if opts.extractaudio:
-        if opts.audioformat not in ['best', 'aac', 'flac', 'mp3', 'm4a', 'opus', 'vorbis', 'wav']:
+        if opts.audioformat not in ['best'] + list(FFmpegExtractAudioPP.SUPPORTED_EXTS):
             parser.error('invalid audio format specified')
     if opts.audioquality:
         opts.audioquality = opts.audioquality.strip('k').strip('K')
         if not opts.audioquality.isdigit():
             parser.error('invalid audio quality specified')
     if opts.recodevideo is not None:
-        if opts.recodevideo not in REMUX_EXTENSIONS:
+        opts.recodevideo = opts.recodevideo.replace(' ', '')
-            parser.error('invalid video recode format specified')
+        if not re.match(FFmpegVideoConvertorPP.FORMAT_RE, opts.recodevideo):
+            parser.error('invalid video remux format specified')
     if opts.remuxvideo is not None:
         opts.remuxvideo = opts.remuxvideo.replace(' ', '')
-        remux_regex = r'{0}(?:/{0})*$'.format(r'(?:\w+>)?(?:%s)' % '|'.join(REMUX_EXTENSIONS))
+        if not re.match(FFmpegVideoRemuxerPP.FORMAT_RE, opts.remuxvideo):
-        if not re.match(remux_regex, opts.remuxvideo):
             parser.error('invalid video remux format specified')
     if opts.convertsubtitles is not None:
-        if opts.convertsubtitles not in ('srt', 'vtt', 'ass', 'lrc'):
+        if opts.convertsubtitles not in FFmpegSubtitlesConvertorPP.SUPPORTED_EXTS:
             parser.error('invalid subtitle format specified')
     if opts.convertthumbnails is not None:
-        if opts.convertthumbnails not in ('jpg', ):
+        if opts.convertthumbnails not in FFmpegThumbnailsConvertorPP.SUPPORTED_EXTS:
             parser.error('invalid thumbnail format specified')
 
     if opts.date is not None:
@@ -258,9 +265,9 @@ def _real_main(argv=None):
         return parsed_compat_opts
 
     all_compat_opts = [
-        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'multistreams',
+        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'no-playlist-metafiles',
-        'no-playlist-metafiles', 'no-live-chat', 'playlist-index', 'list-formats',
+        'multistreams', 'no-live-chat', 'playlist-index', 'list-formats', 'no-direct-merge',
-        'no-youtube-channel-redirect', 'no-youtube-unavailable-videos',
+        'no-youtube-channel-redirect', 'no-youtube-unavailable-videos', 'no-attach-info-json',
     ]
     compat_opts = parse_compat_opts()
 
@@ -301,6 +308,16 @@ def _real_main(argv=None):
         else:
             _unused_compat_opt('filename')
 
+    def validate_outtmpl(tmpl, msg):
+        err = YoutubeDL.validate_outtmpl(tmpl)
+        if err:
+            parser.error('invalid %s %r: %s' % (msg, tmpl, error_to_compat_str(err)))
+
+    for k, tmpl in opts.outtmpl.items():
+        validate_outtmpl(tmpl, '%s output template' % k)
+    for tmpl in opts.forceprint:
+        validate_outtmpl(tmpl, 'print template')
+
     if opts.extractaudio and not opts.keepvideo and opts.format is None:
         opts.format = 'bestaudio/best'
 
@@ -321,7 +338,7 @@ def _real_main(argv=None):
         if re.match(MetadataFromFieldPP.regex, f) is None:
             parser.error('invalid format string "%s" specified for --parse-metadata' % f)
 
-    any_getting = opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
+    any_getting = opts.forceprint or opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
     any_printing = opts.print_json
     download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive
 
@@ -480,10 +497,10 @@ def _real_main(argv=None):
         opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
 
     final_ext = (
-        opts.recodevideo
+        opts.recodevideo if opts.recodevideo in FFmpegVideoConvertorPP.SUPPORTED_EXTS
-        or (opts.remuxvideo in REMUX_EXTENSIONS) and opts.remuxvideo
+        else opts.remuxvideo if opts.remuxvideo in FFmpegVideoRemuxerPP.SUPPORTED_EXTS
-        or (opts.extractaudio and opts.audioformat != 'best') and opts.audioformat
+        else opts.audioformat if (opts.extractaudio and opts.audioformat != 'best')
-        or None)
+        else None)
 
     match_filter = (
         None if opts.match_filter is None
@@ -508,6 +525,7 @@ def _real_main(argv=None):
         'forceduration': opts.getduration,
         'forcefilename': opts.getfilename,
         'forceformat': opts.getformat,
+        'forceprint': opts.forceprint,
         'forcejson': opts.dumpjson or opts.print_json,
         'dump_single_json': opts.dump_single_json,
         'force_write_download_archive': opts.force_write_download_archive,
@@ -656,7 +674,7 @@ def _real_main(argv=None):
         # Update version
         if opts.update_self:
             # If updater returns True, exit. Required for windows
-            if update_self(ydl.to_screen, opts.verbose, ydl._opener):
+            if run_update(ydl):
                 if actual_use:
                     sys.exit('ERROR: The program must exit for the update to complete')
                 sys.exit()

yt_dlp/__main__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Execute with

yt_dlp/downloader/common.py

@@ -164,6 +164,9 @@ class FileDownloader(object):
     def report_error(self, *args, **kargs):
         self.ydl.report_error(*args, **kargs)
 
+    def write_debug(self, *args, **kargs):
+        self.ydl.write_debug(*args, **kargs)
+
     def slow_down(self, start_time, now, byte_counter):
         """Sleep if the download speed is over the rate limit."""
         rate_limit = self.params.get('ratelimit')
@@ -402,5 +405,4 @@ class FileDownloader(object):
         if exe is None:
             exe = os.path.basename(str_args[0])
 
-        self.to_screen('[debug] %s command line: %s' % (
+        self.write_debug('%s command line: %s' % (exe, shell_quote(str_args)))
-            exe, shell_quote(str_args)))

yt_dlp/downloader/dash.py

@@ -154,8 +154,9 @@ class DashSegmentsFD(FragmentFD):
             max_workers = self.params.get('concurrent_fragment_downloads', 1)
             if can_threaded_download and max_workers > 1:
                 self.report_warning('The download speed shown is only of one thread. This is a known issue')
+                _download_fragment = lambda f: (f, download_fragment(f)[1])
                 with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
-                    futures = [pool.submit(download_fragment, fragment) for fragment in fragments_to_download]
+                    futures = [pool.submit(_download_fragment, fragment) for fragment in fragments_to_download]
                     # timeout must be 0 to return instantly
                     done, not_done = concurrent.futures.wait(futures, timeout=0)
                     try:
@@ -169,9 +170,13 @@ class DashSegmentsFD(FragmentFD):
                         # timeout must be none to cancel
                         concurrent.futures.wait(not_done, timeout=None)
                         raise KeyboardInterrupt
-                results = [future.result() for future in futures]
 
-                for frag_content, frag_index in results:
+                for fragment, frag_index in map(lambda x: x.result(), futures):
+                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
+                    down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
+                    fragment['fragment_filename_sanitized'] = frag_sanitized
+                    frag_content = down.read()
+                    down.close()
                     result = append_fragment(frag_content, frag_index)
                     if not result:
                         return False

yt_dlp/downloader/external.py

@@ -346,7 +346,7 @@ class FFmpegFD(ExternalFD):
         return FFmpegPostProcessor().available
 
     def _call_downloader(self, tmpfilename, info_dict):
-        url = info_dict['url']
+        urls = [f['url'] for f in info_dict.get('requested_formats', [])] or [info_dict['url']]
         ffpp = FFmpegPostProcessor(downloader=self)
         if not ffpp.available:
             self.report_error('m3u8 download detected but ffmpeg could not be found. Please install')
@@ -359,6 +359,8 @@ class FFmpegFD(ExternalFD):
             if self.params.get(log_level, False):
                 args += ['-loglevel', log_level]
                 break
+        if not self.params.get('verbose'):
+            args += ['-hide_banner']
 
         seekable = info_dict.get('_seekable')
         if seekable is not None:
@@ -378,7 +380,7 @@ class FFmpegFD(ExternalFD):
         # if end_time:
         #     args += ['-t', compat_str(end_time - start_time)]
 
-        if info_dict.get('http_headers') is not None and re.match(r'^https?://', url):
+        if info_dict.get('http_headers') is not None and re.match(r'^https?://', urls[0]):
             # Trailing \r\n after each HTTP header is important to prevent warning from ffmpeg/avconv:
             # [http @ 00000000003d2fa0] No trailing CRLF found in HTTP header.
             headers = handle_youtubedl_headers(info_dict['http_headers'])
@@ -436,7 +438,15 @@ class FFmpegFD(ExternalFD):
             elif isinstance(conn, compat_str):
                 args += ['-rtmp_conn', conn]
 
-        args += ['-i', url, '-c', 'copy']
+        for url in urls:
+            args += ['-i', url]
+        args += ['-c', 'copy']
+        if info_dict.get('requested_formats'):
+            for (i, fmt) in enumerate(info_dict['requested_formats']):
+                if fmt.get('acodec') != 'none':
+                    args.extend(['-map', '%d:a:0' % i])
+                if fmt.get('vcodec') != 'none':
+                    args.extend(['-map', '%d:v:0' % i])
 
         if self.params.get('test', False):
             args += ['-fs', compat_str(self._TEST_FILE_SIZE)]

yt_dlp/downloader/hls.py

@@ -81,8 +81,6 @@ class HlsFD(FragmentFD):
         man_url = info_dict['url']
         self.to_screen('[%s] Downloading m3u8 manifest' % self.FD_NAME)
 
-        is_webvtt = info_dict['ext'] == 'vtt'
-
         urlh = self.ydl.urlopen(self._prepare_url(info_dict, man_url))
         man_url = urlh.geturl()
         s = urlh.read().decode('utf-8', 'ignore')
@@ -101,6 +99,10 @@ class HlsFD(FragmentFD):
             #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
+        is_webvtt = info_dict['ext'] == 'vtt'
+        if is_webvtt:
+            real_downloader = None  # Packing the fragments is not currently supported for external downloader
+        else:
             real_downloader = _get_real_downloader(info_dict, 'm3u8_frag_urls', self.params, None)
         if real_downloader and not real_downloader.supports_manifest(s):
             real_downloader = None
@@ -270,12 +272,24 @@ class HlsFD(FragmentFD):
             if not success:
                 return False
         else:
+            def decrypt_fragment(fragment, frag_content):
+                decrypt_info = fragment['decrypt_info']
+                if decrypt_info['METHOD'] != 'AES-128':
+                    return frag_content
+                iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', fragment['media_sequence'])
+                decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
+                    self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+                # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
+                # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
+                # not what it decrypts to.
+                if test:
+                    return frag_content
+                return AES.new(decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
+
             def download_fragment(fragment):
                 frag_index = fragment['frag_index']
                 frag_url = fragment['url']
-                decrypt_info = fragment['decrypt_info']
                 byte_range = fragment['byte_range']
-                media_sequence = fragment['media_sequence']
 
                 ctx['fragment_index'] = frag_index
 
@@ -303,18 +317,7 @@ class HlsFD(FragmentFD):
                     self.report_error('Giving up after %s fragment retries' % fragment_retries)
                     return False, frag_index
 
-                if decrypt_info['METHOD'] == 'AES-128':
+                return decrypt_fragment(fragment, frag_content), frag_index
-                    iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
-                    decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                        self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                    # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
-                    # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
-                    # not what it decrypts to.
-                    if not test:
-                        frag_content = AES.new(
-                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
-
-                return frag_content, frag_index
 
             pack_fragment = lambda frag_content, _: frag_content
 
@@ -422,8 +425,9 @@ class HlsFD(FragmentFD):
             max_workers = self.params.get('concurrent_fragment_downloads', 1)
             if can_threaded_download and max_workers > 1:
                 self.report_warning('The download speed shown is only of one thread. This is a known issue')
+                _download_fragment = lambda f: (f, download_fragment(f)[1])
                 with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
-                    futures = [pool.submit(download_fragment, fragment) for fragment in fragments]
+                    futures = [pool.submit(_download_fragment, fragment) for fragment in fragments]
                     # timeout must be 0 to return instantly
                     done, not_done = concurrent.futures.wait(futures, timeout=0)
                     try:
@@ -437,10 +441,14 @@ class HlsFD(FragmentFD):
                         # timeout must be none to cancel
                         concurrent.futures.wait(not_done, timeout=None)
                         raise KeyboardInterrupt
-                results = [future.result() for future in futures]
 
-                for frag_content, frag_index in results:
+                for fragment, frag_index in map(lambda x: x.result(), futures):
-                    result = append_fragment(frag_content, frag_index)
+                    fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], frag_index)
+                    down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
+                    fragment['fragment_filename_sanitized'] = frag_sanitized
+                    frag_content = down.read()
+                    down.close()
+                    result = append_fragment(decrypt_fragment(fragment, frag_content), frag_index)
                     if not result:
                         return False
             else:

yt_dlp/extractor/adobepass.py

@@ -1414,7 +1414,7 @@ class AdobePassIE(InfoExtractor):
                 authn_token = None
             if not authn_token:
                 # TODO add support for other TV Providers
-                mso_id = self._downloader.params.get('ap_mso')
+                mso_id = self.get_param('ap_mso')
                 if not mso_id:
                     raise_mvpd_required()
                 username, password = self._get_login_info('ap_username', 'ap_password', mso_id)

yt_dlp/extractor/afreecatv.py

@@ -323,7 +323,7 @@ class AfreecaTVIE(InfoExtractor):
                         'url': file_url,
                         'format_id': 'http',
                     }]
-                if not formats and not self._downloader.params.get('ignore_no_formats'):
+                if not formats and not self.get_param('ignore_no_formats'):
                     continue
                 self._sort_formats(formats)
                 file_info = common_entry.copy()

yt_dlp/extractor/archiveorg.py

@@ -1,22 +1,36 @@
+# coding: utf-8
 from __future__ import unicode_literals
 
 import re
 import json
 
 from .common import InfoExtractor
-from ..compat import compat_urllib_parse_unquote_plus
+from .youtube import YoutubeIE
+from ..compat import (
+    compat_urllib_parse_unquote,
+    compat_urllib_parse_unquote_plus,
+    compat_urlparse,
+    compat_parse_qs,
+    compat_HTTPError
+)
 from ..utils import (
-    KNOWN_EXTENSIONS,
+    clean_html,
-
+    determine_ext,
+    dict_get,
     extract_attributes,
+    ExtractorError,
+    HEADRequest,
+    int_or_none,
+    KNOWN_EXTENSIONS,
+    merge_dicts,
+    mimetype2ext,
+    parse_duration,
+    RegexNotFoundError,
+    str_to_int,
+    str_or_none,
+    try_get,
     unified_strdate,
     unified_timestamp,
-    clean_html,
-    dict_get,
-    parse_duration,
-    int_or_none,
-    str_or_none,
-    merge_dicts,
 )
 
 
@@ -241,3 +255,165 @@ class ArchiveOrgIE(InfoExtractor):
                     'parent': 'root'})
 
         return info
+
+
+class YoutubeWebArchiveIE(InfoExtractor):
+    IE_NAME = 'web.archive:youtube'
+    IE_DESC = 'web.archive.org saved youtube videos'
+    _VALID_URL = r"""(?x)^
+                (?:https?://)?web\.archive\.org/
+                    (?:web/)?
+                    (?:[0-9A-Za-z_*]+/)?  # /web and the version index is optional
+
+                (?:https?(?::|%3[Aa])//)?
+                (?:
+                    (?:\w+\.)?youtube\.com/watch(?:\?|%3[fF])(?:[^\#]+(?:&|%26))?v(?:=|%3[dD])  # Youtube URL
+                    |(wayback-fakeurl\.archive\.org/yt/)  # Or the internal fake url
+                )
+                (?P<id>[0-9A-Za-z_-]{11})(?:%26|\#|&|$)
+                """
+
+    _TESTS = [
+        {
+            'url': 'https://web.archive.org/web/20150415002341/https://www.youtube.com/watch?v=aYAGB11YrSs',
+            'info_dict': {
+                'id': 'aYAGB11YrSs',
+                'ext': 'webm',
+                'title': 'Team Fortress 2 - Sandviches!'
+            }
+        },
+        {
+            # Internal link
+            'url': 'https://web.archive.org/web/2oe/http://wayback-fakeurl.archive.org/yt/97t7Xj_iBv0',
+            'info_dict': {
+                'id': '97t7Xj_iBv0',
+                'ext': 'mp4',
+                'title': 'How Flexible Machines Could Save The World'
+            }
+        },
+        {
+            # Video from 2012, webm format itag 45.
+            'url': 'https://web.archive.org/web/20120712231619/http://www.youtube.com/watch?v=AkhihxRKcrs&gl=US&hl=en',
+            'info_dict': {
+                'id': 'AkhihxRKcrs',
+                'ext': 'webm',
+                'title': 'Limited Run: Mondo\'s Modern Classic 1 of 3 (SDCC 2012)'
+            }
+        },
+        {
+            # Old flash-only video. Webpage title starts with "YouTube - ".
+            'url': 'https://web.archive.org/web/20081211103536/http://www.youtube.com/watch?v=jNQXAC9IVRw',
+            'info_dict': {
+                'id': 'jNQXAC9IVRw',
+                'ext': 'unknown_video',
+                'title': 'Me at the zoo'
+            }
+        },
+        {
+            # Flash video with .flv extension (itag 34). Title has prefix "YouTube         -"
+            # Title has some weird unicode characters too.
+            'url': 'https://web.archive.org/web/20110712231407/http://www.youtube.com/watch?v=lTx3G6h2xyA',
+            'info_dict': {
+                'id': 'lTx3G6h2xyA',
+                'ext': 'flv',
+                'title': '‪Madeon - Pop Culture (live mashup)‬‏'
+            }
+        },
+        {   # Some versions of Youtube have have "YouTube" as page title in html (and later rewritten by js).
+            'url': 'https://web.archive.org/web/http://www.youtube.com/watch?v=kH-G_aIBlFw',
+            'info_dict': {
+                'id': 'kH-G_aIBlFw',
+                'ext': 'mp4',
+                'title': 'kH-G_aIBlFw'
+            },
+            'expected_warnings': [
+                'unable to extract title',
+            ]
+        },
+        {
+            # First capture is a 302 redirect intermediary page.
+            'url': 'https://web.archive.org/web/20050214000000/http://www.youtube.com/watch?v=0altSZ96U4M',
+            'info_dict': {
+                'id': '0altSZ96U4M',
+                'ext': 'mp4',
+                'title': '0altSZ96U4M'
+            },
+            'expected_warnings': [
+                'unable to extract title',
+            ]
+        },
+        {
+            # Video not archived, only capture is unavailable video page
+            'url': 'https://web.archive.org/web/20210530071008/https://www.youtube.com/watch?v=lHJTf93HL1s&spfreload=10',
+            'only_matching': True,
+        },
+        {   # Encoded url
+            'url': 'https://web.archive.org/web/20120712231619/http%3A//www.youtube.com/watch%3Fgl%3DUS%26v%3DAkhihxRKcrs%26hl%3Den',
+            'only_matching': True,
+        },
+        {
+            'url': 'https://web.archive.org/web/20120712231619/http%3A//www.youtube.com/watch%3Fv%3DAkhihxRKcrs%26gl%3DUS%26hl%3Den',
+            'only_matching': True,
+        }
+    ]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        title = video_id  # if we are not able get a title
+
+        def _extract_title(webpage):
+            page_title = self._html_search_regex(
+                r'<title>([^<]*)</title>', webpage, 'title', fatal=False) or ''
+            # YouTube video pages appear to always have either 'YouTube -' as suffix or '- YouTube' as prefix.
+            try:
+                page_title = self._html_search_regex(
+                    r'(?:YouTube\s*-\s*(.*)$)|(?:(.*)\s*-\s*YouTube$)',
+                    page_title, 'title', default='')
+            except RegexNotFoundError:
+                page_title = None
+
+            if not page_title:
+                self.report_warning('unable to extract title', video_id=video_id)
+                return
+            return page_title
+
+        # If the video is no longer available, the oldest capture may be one before it was removed.
+        # Setting the capture date in url to early date seems to redirect to earliest capture.
+        webpage = self._download_webpage(
+            'https://web.archive.org/web/20050214000000/http://www.youtube.com/watch?v=%s' % video_id,
+            video_id=video_id, fatal=False, errnote='unable to download video webpage (probably not archived).')
+        if webpage:
+            title = _extract_title(webpage) or title
+
+        # Use link translator mentioned in https://github.com/ytdl-org/youtube-dl/issues/13655
+        internal_fake_url = 'https://web.archive.org/web/2oe_/http://wayback-fakeurl.archive.org/yt/%s' % video_id
+        try:
+            video_file_webpage = self._request_webpage(
+                HEADRequest(internal_fake_url), video_id,
+                note='Fetching video file url', expected_status=True)
+        except ExtractorError as e:
+            # HTTP Error 404 is expected if the video is not saved.
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 404:
+                raise ExtractorError(
+                    'HTTP Error %s. Most likely the video is not archived or issue with web.archive.org.' % e.cause.code,
+                    expected=True)
+            raise
+        video_file_url = compat_urllib_parse_unquote(video_file_webpage.url)
+        video_file_url_qs = compat_parse_qs(compat_urlparse.urlparse(video_file_url).query)
+
+        # Attempt to recover any ext & format info from playback url
+        format = {'url': video_file_url}
+        itag = try_get(video_file_url_qs, lambda x: x['itag'][0])
+        if itag and itag in YoutubeIE._formats:  # Naughty access but it works
+            format.update(YoutubeIE._formats[itag])
+            format.update({'format_id': itag})
+        else:
+            mime = try_get(video_file_url_qs, lambda x: x['mime'][0])
+            ext = mimetype2ext(mime) or determine_ext(video_file_url)
+            format.update({'ext': ext})
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': [format],
+            'duration': str_to_int(try_get(video_file_url_qs, lambda x: x['dur'][0]))
+        }

yt_dlp/extractor/ard.py

@@ -290,14 +290,14 @@ class ARDMediathekIE(ARDMediathekBaseIE):
 
 
 class ARDIE(InfoExtractor):
-    _VALID_URL = r'(?P<mainurl>https?://(?:www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?:video-?)?(?P<id>[0-9]+))\.html'
+    _VALID_URL = r'(?P<mainurl>https?://(?:www\.)?daserste\.de/(?:[^/?#&]+/)+(?P<id>[^/?#&]+))\.html'
     _TESTS = [{
         # available till 7.01.2022
         'url': 'https://www.daserste.de/information/talk/maischberger/videos/maischberger-die-woche-video100.html',
         'md5': '867d8aa39eeaf6d76407c5ad1bb0d4c1',
         'info_dict': {
-            'display_id': 'maischberger-die-woche',
+            'id': 'maischberger-die-woche-video100',
-            'id': '100',
+            'display_id': 'maischberger-die-woche-video100',
             'ext': 'mp4',
             'duration': 3687.0,
             'title': 'maischberger. die woche vom 7. Januar 2021',
@@ -305,16 +305,28 @@ class ARDIE(InfoExtractor):
             'thumbnail': r're:^https?://.*\.jpg$',
         },
     }, {
-        'url': 'https://www.daserste.de/information/reportage-dokumentation/erlebnis-erde/videosextern/woelfe-und-herdenschutzhunde-ungleiche-brueder-102.html',
+        'url': 'https://www.daserste.de/information/politik-weltgeschehen/morgenmagazin/videosextern/dominik-kahun-aus-der-nhl-direkt-zur-weltmeisterschaft-100.html',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.daserste.de/information/nachrichten-wetter/tagesthemen/videosextern/tagesthemen-17736.html',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.daserste.de/unterhaltung/serie/in-aller-freundschaft-die-jungen-aerzte/videos/diversity-tag-sanam-afrashteh100.html',
         'only_matching': True,
     }, {
         'url': 'http://www.daserste.de/information/reportage-dokumentation/dokus/videos/die-story-im-ersten-mission-unter-falscher-flagge-100.html',
         'only_matching': True,
+    }, {
+        'url': 'https://www.daserste.de/unterhaltung/serie/in-aller-freundschaft-die-jungen-aerzte/Drehpause-100.html',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.daserste.de/unterhaltung/film/filmmittwoch-im-ersten/videos/making-ofwendezeit-video-100.html',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
-        display_id = mobj.group('display_id')
+        display_id = mobj.group('id')
 
         player_url = mobj.group('mainurl') + '~playerXml.xml'
         doc = self._download_xml(player_url, display_id)
@@ -365,7 +377,7 @@ class ARDIE(InfoExtractor):
         self._sort_formats(formats)
 
         return {
-            'id': mobj.group('id'),
+            'id': xpath_text(video_node, './videoId', default=display_id),
             'formats': formats,
             'display_id': display_id,
             'title': video_node.find('./title').text,

yt_dlp/extractor/audius.py

@@ -245,3 +245,31 @@ class AudiusPlaylistIE(AudiusBaseIE):
         return self.playlist_result(entries, playlist_id,
                                     playlist_data.get('playlist_name', title),
                                     playlist_data.get('description'))
+
+
+class AudiusProfileIE(AudiusPlaylistIE):
+    IE_NAME = 'audius:artist'
+    IE_DESC = 'Audius.co profile/artist pages'
+    _VALID_URL = r'https?://(?:www)?audius\.co/(?P<id>[^\/]+)/?(?:[?#]|$)'
+    _TEST = {
+        'url': 'https://audius.co/pzl/',
+        'info_dict': {
+            'id': 'ezRo7',
+            'description': 'TAMALE\n\nContact: officialpzl@gmail.com',
+            'title': 'pzl',
+        },
+        'playlist_count': 24,
+    }
+
+    def _real_extract(self, url):
+        self._select_api_base()
+        profile_id = self._match_id(url)
+        try:
+            _profile_data = self._api_request('/full/users/handle/' + profile_id, profile_id)
+        except ExtractorError as e:
+            raise ExtractorError('Could not download profile info; ' + str(e))
+        profile_audius_id = _profile_data[0]['id']
+        profile_bio = _profile_data[0].get('bio')
+
+        api_call = self._api_request('/full/users/handle/%s/tracks' % profile_id, profile_id)
+        return self.playlist_result(self._build_playlist(api_call), profile_audius_id, profile_id, profile_bio)