Release 2021.01.29

* Refactor update-version
* Moved pyinst, update-version and icon into devscripts
* pyinst doesn't bump version anymore
* Merge pyinst and pyinst32. Usage: `pyinst.py [32|64]`
* Add mutagen as requirement
* Remove make_win and related files

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.10
+ [debug] yt-dlp version 2021.01.24
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,16 +21,16 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
-- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlc
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
+- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlp
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.10
+ [debug] yt-dlp version 2021.01.24
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.24. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.24**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/6_question.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- Look through the README (https://github.com/pukkandan/yt-dlc) and FAQ (https://github.com/pukkandan/yt-dlc) for similar questions
+- Look through the README (https://github.com/pukkandan/yt-dlp) and FAQ (https://github.com/pukkandan/yt-dlp) for similar questions
 - Search the bugtracker for similar questions: https://github.com/blackjack4494/yt-dlc
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -21,10 +21,10 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -21,10 +21,10 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md

@@ -21,8 +21,8 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -21,11 +21,11 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
-- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
-- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlc
+- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
+- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
+- Read bugs section in FAQ: https://github.com/pukkandan/yt-dlp
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md

@@ -21,8 +21,8 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -8,7 +8,7 @@
 
 ### Before submitting a *pull request* make sure you have:
 - [ ] At least skimmed through [adding new extractor tutorial](https://github.com/ytdl-org/youtube-dl#adding-support-for-a-new-site) and [youtube-dl coding conventions](https://github.com/ytdl-org/youtube-dl#youtube-dl-coding-conventions) sections
-- [ ] [Searched](https://github.com/pukkandan/yt-dlc/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
+- [ ] [Searched](https://github.com/pukkandan/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
 - [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)
 
 ### In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:

.github/workflows/build.yml

@@ -25,8 +25,8 @@ jobs:
       run: sudo apt-get -y install zip pandoc man
     - name: Bump version
       id: bump_version
-      run: python scripts/update-version-workflow.py
-    - name: Check the output from My action
+      run: python devscripts/update-version.py
+    - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run Make
       run: make
@@ -37,7 +37,7 @@ jobs:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       with:
         tag_name: ${{ steps.bump_version.outputs.ytdlc_version }}
-        release_name: youtube-dlc ${{ steps.bump_version.outputs.ytdlc_version }}
+        release_name: yt-dlp ${{ steps.bump_version.outputs.ytdlc_version }}
         body: |
           Changelog:
           PLACEHOLDER
@@ -58,18 +58,18 @@ jobs:
       env:
         SHA2: ${{ hashFiles('youtube-dlc') }}
       run: echo "::set-output name=sha2_unix::$SHA2"
-    # - name: Install dependencies for pypi
-    #   run: |
-    #     python -m pip install --upgrade pip
-    #     pip install setuptools wheel twine
-    # - name: Build and publish
-    #   env:
-    #     TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-    #     TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
-    #   run: |
-    #     rm -rf dist/*
-    #     python setup.py sdist bdist_wheel
-    #     twine upload dist/*
+    - name: Install dependencies for pypi
+      run: |
+        python -m pip install --upgrade pip
+        pip install setuptools wheel twine
+    - name: Build and publish on pypi
+      env:
+        TWINE_USERNAME: __token__
+        TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+      run: |
+        rm -rf dist/*
+        python setup.py sdist bdist_wheel
+        twine upload dist/*
 
   build_windows:
 
@@ -84,11 +84,14 @@ jobs:
       with:
           python-version: '3.8'
     - name: Install Requirements
-      run: pip install pyinstaller
+      run: pip install pyinstaller mutagen
     - name: Bump version
-      run: python scripts/update-version-workflow.py
+      id: bump_version
+      run: python devscripts/update-version.py
+    - name: Print version
+      run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script
-      run: python pyinst.py
+      run: python devscripts/pyinst.py 64
     - name: Upload youtube-dlc.exe Windows binary
       id: upload-release-windows
       uses: actions/upload-release-asset@v1
@@ -119,11 +122,14 @@ jobs:
           python-version: '3.4.4'
           architecture: 'x86'
     - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5
+      run: pip install pyinstaller==3.5 mutagen
     - name: Bump version
-      run: python scripts/update-version-workflow.py
+      id: bump_version
+      run: python devscripts/update-version.py
+    - name: Print version
+      run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script for 32 Bit
-      run: python pyinst32.py
+      run: python devscripts/pyinst.py 32
     - name: Upload Executable youtube-dlc_x86.exe
       id: upload-release-windows32
       uses: actions/upload-release-asset@v1
@@ -162,18 +168,15 @@ jobs:
         asset_name: SHA2-256SUMS
         asset_content_type: text/plain
 
-  update_version_badge:
-
-    runs-on: ubuntu-latest
-
-    needs: build_unix
-
-    steps:
-    - name: Create Version Badge
-      uses: schneegans/dynamic-badges-action@v1.0.0
-      with:
-        auth: ${{ secrets.GIST_TOKEN }}
-        gistID: c69cb23c3c5b3316248e52022790aa57
-        filename: version.json
-        label: Version
-        message: ${{ needs.build_unix.outputs.ytdlc_version }}
+#  update_version_badge:
+#    runs-on: ubuntu-latest
+#    needs: build_unix
+#    steps:
+#    - name: Create Version Badge
+#      uses: schneegans/dynamic-badges-action@v1.0.0
+#      with:
+#        auth: ${{ secrets.GIST_TOKEN }}
+#        gistID: c69cb23c3c5b3316248e52022790aa57
+#        filename: version.json
+#        label: Version
+#        message: ${{ needs.build_unix.outputs.ytdlc_version }}

.github/workflows/core.yml

@@ -1,9 +1,9 @@
-name: Full Test
-on: [push]
+name: Core Tests
+on: [push, pull_request]
 jobs:
   tests:
-    name: Tests
-    if: "!contains(github.event.head_commit.message, 'skip ci')"
+    name: Core Tests
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: true
@@ -12,7 +12,7 @@ jobs:
         # TODO: python 2.6
         python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
         python-impl: [cpython]
-        ytdl-test-set: [core, download]
+        ytdl-test-set: [core]
         run-tests-ext: [sh]
         include:
         # python 3.2 is only available on windows via setup-python
@@ -21,20 +21,11 @@ jobs:
           python-impl: cpython
           ytdl-test-set: core
           run-tests-ext: bat
-        - os: windows-latest
-          python-version: 3.2
-          python-impl: cpython
-          ytdl-test-set: download
-          run-tests-ext: bat
         # jython
         - os: ubuntu-latest
           python-impl: jython
           ytdl-test-set: core
           run-tests-ext: sh
-        - os: ubuntu-latest
-          python-impl: jython
-          ytdl-test-set: download
-          run-tests-ext: sh
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}
@@ -60,4 +51,4 @@ jobs:
       env:
         YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
       run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
-  # flake8 has been moved to quick-test
+  # Linter is in quick-test

.github/workflows/download.yml

@@ -0,0 +1,53 @@
+name: Download Tests
+on: [push, pull_request]
+jobs:
+  tests:
+    name: Download Tests
+    if: "!contains(github.event.head_commit.message, 'ci skip dl') && !contains(github.event.head_commit.message, 'ci skip all')"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: true
+      matrix:
+        os: [ubuntu-18.04]
+        # TODO: python 2.6
+        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
+        python-impl: [cpython]
+        ytdl-test-set: [download]
+        run-tests-ext: [sh]
+        include:
+        # python 3.2 is only available on windows via setup-python
+        - os: windows-latest
+          python-version: 3.2
+          python-impl: cpython
+          ytdl-test-set: download
+          run-tests-ext: bat
+        # jython - disable for now since it takes too long to complete
+        # - os: ubuntu-latest
+        #   python-impl: jython
+        #   ytdl-test-set: download
+        #   run-tests-ext: sh
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      if: ${{ matrix.python-impl == 'cpython' }}
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Set up Java 8
+      if: ${{ matrix.python-impl == 'jython' }}
+      uses: actions/setup-java@v1
+      with:
+        java-version: 8
+    - name: Install Jython
+      if: ${{ matrix.python-impl == 'jython' }}
+      run: |
+        wget http://search.maven.org/remotecontent?filepath=org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
+        java -jar jython-installer.jar -s -d "$HOME/jython"
+        echo "$HOME/jython/bin" >> $GITHUB_PATH
+    - name: Install nose
+      run: pip install nose
+    - name: Run tests
+      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
+      env:
+        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
+      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}

.github/workflows/python-publish.yml.disable

@@ -1,33 +0,0 @@
-# This workflows will upload a Python Package using Twine when a release is created
-# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
-
-name: Upload Python Package
-
-on:
-  push:
-    branches:
-      - release
-
-jobs:
-  deploy:
-
-    runs-on: ubuntu-latest
-
-    steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: '3.x'
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install setuptools wheel twine
-    - name: Build and publish
-      env:
-        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
-      run: |
-        rm -rf dist/*
-        python setup.py sdist bdist_wheel
-        twine upload dist/*

.github/workflows/quick-test.yml

@@ -1,13 +1,13 @@
-name: Core Test
-on: [push]
+name: Quick Test
+on: [push, pull_request]
 jobs:
   tests:
     name: Core Tests
-    if: "!contains(github.event.head_commit.message, 'skip ci all')"
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
-    - name: Set up Python 3.9
+    - name: Set up Python
       uses: actions/setup-python@v2
       with:
         python-version: 3.9
@@ -19,7 +19,7 @@ jobs:
       run: ./devscripts/run_tests.sh
   flake8:
     name: Linter
-    if: "!contains(github.event.head_commit.message, 'skip ci all')"
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2

.gitignore

@@ -1,35 +1,43 @@
+# Python
 *.pyc
 *.pyo
-*.class
-*~
-*.DS_Store
 wine-py2exe/
 py2exe.log
-*.kate-swp
 build/
 dist/
 zip/
+tmp/
+venv/
+
+# Misc
+*~
+*.DS_Store
+*.kate-swp
 MANIFEST
-README.txt
-youtube-dl.1
-youtube-dlc.1
-youtube-dl.bash-completion
-youtube-dlc.bash-completion
-youtube-dl.fish
-youtube-dlc.fish
-youtube_dl/extractor/lazy_extractors.py
-youtube_dlc/extractor/lazy_extractors.py
-youtube-dl
-youtube-dlc
-youtube-dl.exe
-youtube-dlc.exe
-youtube-dl.tar.gz
-youtube-dlc.tar.gz
-youtube-dlc.spec
+test/local_parameters.json
 .coverage
 cover/
 updates_key.pem
 *.egg-info
+.tox
+*.class
+
+# Generated
+README.txt
+*.1
+*.bash-completion
+*.fish
+*.exe
+*.tar.gz
+*.zsh
+*.spec
+
+# Binary
+youtube-dl
+youtube-dlc
+*.exe
+
+# Downloaded
 *.srt
 *.ttml
 *.sbv
@@ -47,22 +55,29 @@ updates_key.pem
 *.part
 *.ytdl
 *.swp
+*.ogg
+*.opus
+
+# Config
+*.conf
 *.spec
-test/local_parameters.json
-.tox
-youtube-dl.zsh
-youtube-dlc.zsh
-
-# IntelliJ related files
-.idea
-*.iml
-
-tmp/
-venv/
-
-# VS Code related files
-.vscode
-
+cookies
 cookies.txt
 
-*.sublime-workspace
+
+
+# Text Editor / IDE
+.idea
+*.iml
+.vscode
+*.sublime-workspace
+*.sublime-project
+!yt-dlp.sublime-project
+
+# Lazy extractors
+*/extractor/lazy_extractors.py
+
+# Plugins
+ytdlp_plugins/extractor/*
+!ytdlp_plugins/extractor/__init__.py
+!ytdlp_plugins/extractor/sample.py

AUTHORS

@@ -1,248 +0,0 @@
-Ricardo Garcia Gonzalez
-Danny Colligan
-Benjamin Johnson
-Vasyl' Vavrychuk
-Witold Baryluk
-Paweł Paprota
-Gergely Imreh
-Rogério Brito
-Philipp Hagemeister
-Sören Schulze
-Kevin Ngo
-Ori Avtalion
-shizeeg
-Filippo Valsorda
-Christian Albrecht
-Dave Vasilevsky
-Jaime Marquínez Ferrándiz
-Jeff Crouse
-Osama Khalid
-Michael Walter
-M. Yasoob Ullah Khalid
-Julien Fraichard
-Johny Mo Swag
-Axel Noack
-Albert Kim
-Pierre Rudloff
-Huarong Huo
-Ismael Mejía
-Steffan Donal
-Andras Elso
-Jelle van der Waa
-Marcin Cieślak
-Anton Larionov
-Takuya Tsuchida
-Sergey M.
-Michael Orlitzky
-Chris Gahan
-Saimadhav Heblikar
-Mike Col
-Oleg Prutz
-pulpe
-Andreas Schmitz
-Michael Kaiser
-Niklas Laxström
-David Triendl
-Anthony Weems
-David Wagner
-Juan C. Olivares
-Mattias Harrysson
-phaer
-Sainyam Kapoor
-Nicolas Évrard
-Jason Normore
-Hoje Lee
-Adam Thalhammer
-Georg Jähnig
-Ralf Haring
-Koki Takahashi
-Ariset Llerena
-Adam Malcontenti-Wilson
-Tobias Bell
-Naglis Jonaitis
-Charles Chen
-Hassaan Ali
-Dobrosław Żybort
-David Fabijan
-Sebastian Haas
-Alexander Kirk
-Erik Johnson
-Keith Beckman
-Ole Ernst
-Aaron McDaniel (mcd1992)
-Magnus Kolstad
-Hari Padmanaban
-Carlos Ramos
-5moufl
-lenaten
-Dennis Scheiba
-Damon Timm
-winwon
-Xavier Beynon
-Gabriel Schubiner
-xantares
-Jan Matějka
-Mauroy Sébastien
-William Sewell
-Dao Hoang Son
-Oskar Jauch
-Matthew Rayfield
-t0mm0
-Tithen-Firion
-Zack Fernandes
-cryptonaut
-Adrian Kretz
-Mathias Rav
-Petr Kutalek
-Will Glynn
-Max Reimann
-Cédric Luthi
-Thijs Vermeir
-Joel Leclerc
-Christopher Krooss
-Ondřej Caletka
-Dinesh S
-Johan K. Jensen
-Yen Chi Hsuan
-Enam Mijbah Noor
-David Luhmer
-Shaya Goldberg
-Paul Hartmann
-Frans de Jonge
-Robin de Rooij
-Ryan Schmidt
-Leslie P. Polzer
-Duncan Keall
-Alexander Mamay
-Devin J. Pohly
-Eduardo Ferro Aldama
-Jeff Buchbinder
-Amish Bhadeshia
-Joram Schrijver
-Will W.
-Mohammad Teimori Pabandi
-Roman Le Négrate
-Matthias Küch
-Julian Richen
-Ping O.
-Mister Hat
-Peter Ding
-jackyzy823
-George Brighton
-Remita Amine
-Aurélio A. Heckert
-Bernhard Minks
-sceext
-Zach Bruggeman
-Tjark Saul
-slangangular
-Behrouz Abbasi
-ngld
-nyuszika7h
-Shaun Walbridge
-Lee Jenkins
-Anssi Hannula
-Lukáš Lalinský
-Qijiang Fan
-Rémy Léone
-Marco Ferragina
-reiv
-Muratcan Simsek
-Evan Lu
-flatgreen
-Brian Foley
-Vignesh Venkat
-Tom Gijselinck
-Founder Fang
-Andrew Alexeyew
-Saso Bezlaj
-Erwin de Haan
-Jens Wille
-Robin Houtevelts
-Patrick Griffis
-Aidan Rowe
-mutantmonkey
-Ben Congdon
-Kacper Michajłow
-José Joaquín Atria
-Viťas Strádal
-Kagami Hiiragi
-Philip Huppert
-blahgeek
-Kevin Deldycke
-inondle
-Tomáš Čech
-Déstin Reed
-Roman Tsiupa
-Artur Krysiak
-Jakub Adam Wieczorek
-Aleksandar Topuzović
-Nehal Patel
-Rob van Bekkum
-Petr Zvoníček
-Pratyush Singh
-Aleksander Nitecki
-Sebastian Blunt
-Matěj Cepl
-Xie Yanbo
-Philip Xu
-John Hawkinson
-Rich Leeper
-Zhong Jianxin
-Thor77
-Mattias Wadman
-Arjan Verwer
-Costy Petrisor
-Logan B
-Alex Seiler
-Vijay Singh
-Paul Hartmann
-Stephen Chen
-Fabian Stahl
-Bagira
-Odd Stråbø
-Philip Herzog
-Thomas Christlieb
-Marek Rusinowski
-Tobias Gruetzmacher
-Olivier Bilodeau
-Lars Vierbergen
-Juanjo Benages
-Xiao Di Guan
-Thomas Winant
-Daniel Twardowski
-Jeremie Jarosh
-Gerard Rovira
-Marvin Ewald
-Frédéric Bournival
-Timendum
-gritstub
-Adam Voss
-Mike Fährmann
-Jan Kundrát
-Giuseppe Fabiano
-Örn Guðjónsson
-Parmjit Virk
-Genki Sky
-Ľuboš Katrinec
-Corey Nicholson
-Ashutosh Chaudhary
-John Dong
-Tatsuyuki Ishi
-Daniel Weber
-Kay Bouché
-Yang Hongbo
-Lei Wang
-Petr Novák
-Leonardo Taccari
-Martin Weinelt
-Surya Oktafendri
-TingPing
-Alexandre Macabies
-Bastian de Groot
-Niklas Haas
-András Veres-Szentkirályi
-Enes Solak
-Nathan Rossi
-Thomas van der Berg
-Luca Cherubin

CONTRIBUTORS

@@ -14,4 +14,6 @@ mariuszskon
 ohnonot
 samiksome
 alxnull
-FelixFrog
+FelixFrog
+Zocker1999NET
+nao20010128nao

Changelog.md

@@ -4,20 +4,102 @@
 # Instuctions for creating release
 
 * Run `make doc`
-* Update Changelog.md and Authors-Fork
+* Update Changelog.md and CONTRIBUTORS
+* Change "Merged with youtube-dl" version in Readme.md if needed
 * Commit to master as `Release <version>`
 * Push to origin/release - build task will now run
-* Update version.py and run `make issuetemplates`
-* Commit to master as `[version] update`
+* Update version.py using devscripts\update-version.py
+* Run `make issuetemplates`
+* Commit to master as `[version] update :ci skip all`
 * Push to origin/master
 * Update changelog in /releases
 
 -->
 
 
+### 2021.01.29
+* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by @animelover1984 and @bbepis
+    * Add `--get-comments`
+    * [youtube] Extract comments
+    * [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE
+    * [billibilli] Extract comments
+    * [billibilli] Better video extraction
+    * Write playlist data to infojson
+    * [FFmpegMetadata] Embed infojson inside the video
+    * [EmbedThumbnail] Try embedding in mp4 using ffprobe and `-disposition`
+    * [EmbedThumbnail] Treat mka like mkv and mov like mp4
+    * [EmbedThumbnail] Embed in ogg/opus
+    * [VideoRemuxer] Conditionally remux video
+    * [VideoRemuxer] Add `-movflags +faststart` when remuxing to mp4
+    * [ffmpeg] Print entire stderr in verbose when there is error
+    * [EmbedSubtitle] Warn when embedding ass in mp4
+    * [anvato] Use NFLTokenGenerator if possible
+* **Parse additional metadata**: New option `--parse-metadata` to extract additional metadata from existing fields
+    * The extracted fields can be used in `--output`
+    * Deprecated `--metadata-from-title`
+* [Audius] Add extractor
+* [youtube] Extract playlist description and write it to `.description` file
+* Detect existing files even when using `recode`/`remux` (`extract-audio` is partially fixed)
+* Fix wrong user config from v2021.01.24
+* [youtube] Report error message from youtube as error instead of warning
+* [FormatSort] Fix some fields not sorting from v2021.01.24
+* [postprocessor] Deprecate `avconv`/`avprobe`.  All current functionality is left untouched. But don't expect any new features to work with avconv
+* [postprocessor] fix `write_debug` to not throw error when there is no `_downloader`
+* [movefiles] Don't give "cant find" warning when move is unnecessary
+* Refactor `update-version`, `pyinst.py` and related files
+* [ffmpeg] Document more formats that are supported for remux/recode
+
+
+### 2021.01.24
+* **Merge youtube-dl:** Upto [2021.01.24](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
+* Plugin support ([documentation](https://github.com/pukkandan/yt-dlp#plugins))
+* **Multiple paths**: New option `-P`/`--paths` to give different paths for different types of files
+    * The syntax is `-P "type:path" -P "type:path"` ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=-P,%20--paths%20TYPE:PATH))
+    * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
+    * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=Home%20Configuration))
+* Allow passing different arguments to different external downloaders ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
+* [mildom] Add extractor by @nao20010128nao
+* Warn when using old style `--external-downloader-args` and `--post-processor-args`
+* Fix `--no-overwrite` when using `--write-link`
+* [sponskrub] Output `unrecognized argument` error message correctly
+* [cbs] Make failure to extract title non-fatal
+* Fix typecasting when pre-checking archive
+* Fix issue with setting title on UNIX
+* Deprecate redundant aliases in `formatSort`. The aliases remain functional for backward compatibility, but will be left undocumented
+* [tests] Fix test_post_hooks
+* [tests] Split core and download tests
+
+
+### 2021.01.20
+* [TrovoLive] Add extractor (only VODs)
+* [pokemon] Add `/#/player` URLs
+* Improved parsing of multiple postprocessor-args, add `--ppa` as alias
+* [EmbedThumbnail] Simplify embedding in mkv
+* [sponskrub] Encode filenames correctly, better debug output and error message
+* [readme] Cleanup options
+
+
+### 2021.01.16
+* **Merge youtube-dl:** Upto [2021.01.16](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
+* **Configuration files:**
+    * Portable configuration file: `./yt-dlp.conf`
+    * Allow the configuration files to be named `yt-dlp` instead of `youtube-dlc`. See [this](https://github.com/pukkandan/yt-dlp#configuration) for details
+* Add PyPI release
+
+
+### 2021.01.14
+* Added option `--break-on-reject`
+* [roosterteeth.com] Fix for bonus episodes by @Zocker1999NET
+* [tiktok] Fix for when share_info is empty
+* [EmbedThumbnail] Fix bug due to incorrect function name
+* [documentation] Changed sponskrub links to point to [pukkandan/sponskrub](https://github.com/pukkandan/SponSkrub) since I am now providing both linux and windows releases
+* [documentation] Change all links to correctly point to new fork URL
+* [documentation] Fixes typos
+
+
 ### 2021.01.12
 * [roosterteeth.com] Add subtitle support by @samiksome
-* Added `--force-overwrites` by @alxnull
+* Added `--force-overwrites`, `--no-force-overwrites` by @alxnull
 * Changed fork name to `yt-dlp`
 * Fix typos by @FelixFrog
 * [ci] Option to skip
@@ -44,7 +126,7 @@
 
 
 ### 2021.01.08
-* **Merge youtube-dl:** Upto [2021.01.08](https://github.com/ytdl-org/youtube-dl/commit/bf6a74c620bd4d5726503c5302906bb36b009026)
+* **Merge youtube-dl:** Upto [2021.01.08](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.08)
     * Extractor stitcher ([1](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc), [2](https://github.com/ytdl-org/youtube-dl/commit/a563c97c5cddf55f8989ed7ea8314ef78e30107f)) have not been merged
 * Moved changelog to seperate file
 
@@ -86,7 +168,7 @@
     * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
 * **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by @h-h-h-h - See [Internet Shortcut Options]README.md(#internet-shortcut-options) for details
 * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details
-* Added `--force-download-archive` (`--force-write-archive`) by by h-h-h-h
+* Added `--force-download-archive` (`--force-write-archive`) by @h-h-h-h
 * Added `--list-formats-as-table`,  `--list-formats-old`
 * **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file
     * Added `--no-ignore-dynamic-mpd`, `--no-allow-dynamic-mpd`, `--allow-dynamic-mpd`, `--youtube-include-hls-manifest`, `--no-youtube-include-hls-manifest`, `--no-youtube-skip-hls-manifest`, `--no-download`, `--no-download-archive`, `--resize-buffer`, `--part`, `--mtime`, `--no-keep-fragments`, `--no-cookies`, `--no-write-annotations`, `--no-write-info-json`, `--no-write-description`, `--no-write-thumbnail`, `--youtube-include-dash-manifest`, `--post-overwrites`, `--no-keep-video`, `--no-embed-subs`, `--no-embed-thumbnail`, `--no-add-metadata`, `--no-include-ads`, `--no-write-sub`, `--no-write-auto-sub`, `--no-playlist-reverse`, `--no-restrict-filenames`, `--youtube-include-dash-manifest`, `--no-format-sort-force`, `--flat-videos`, `--no-list-formats-as-table`, `--no-sponskrub`, `--no-sponskrub-cut`, `--no-sponskrub-force`

Makefile

@@ -1,8 +1,10 @@
-all: youtube-dlc README.md CONTRIBUTING.md README.txt issuetemplates youtube-dlc.1 youtube-dlc.bash-completion youtube-dlc.zsh youtube-dlc.fish supportedsites
+all: youtube-dlc doc man
 doc: README.md CONTRIBUTING.md issuetemplates supportedsites
+man: README.txt youtube-dlc.1 youtube-dlc.bash-completion youtube-dlc.zsh youtube-dlc.fish
+
 
 clean:
-	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
+	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
 	find . -name "*.pyc" -delete
 	find . -name "*.class" -delete
 

README.md

@@ -1,10 +1,12 @@
 # YT-DLP
 
-<!-- See: https://github.com/marketplace/actions/dynamic-badges -->
-[![Release Version](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/pukkandan/c69cb23c3c5b3316248e52022790aa57/raw/version.json&color=brightgreen)](https://github.com/pukkandan/yt-dlc/releases/latest)
-[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](https://github.com/pukkandan/yt-dlc/blob/master/LICENSE)
-[![Core Status](https://github.com/pukkandan/yt-dlc/workflows/Core%20Test/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlc/actions?query=workflow%3ACore)
-[![CI Status](https://github.com/pukkandan/yt-dlc/workflows/Full%20Test/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlc/actions?query=workflow%3AFull)
+[![Release version](https://img.shields.io/github/v/release/pukkandan/yt-dlp?color=brightgreen&label=Release)](https://github.com/pukkandan/yt-dlp/releases/latest)
+[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
+[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)  
+[![Commits](https://img.shields.io/github/commit-activity/m/pukkandan/yt-dlp?label=commits)](https://github.com/pukkandan/yt-dlp/commits)
+[![Last Commit](https://img.shields.io/github/last-commit/pukkandan/yt-dlp/master)](https://github.com/pukkandan/yt-dlp/commits)
+[![Downloads](https://img.shields.io/github/downloads/pukkandan/yt-dlp/total)](https://github.com/pukkandan/yt-dlp/releases/latest)
+[![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi)](https://pypi.org/project/yt-dlp)
 
 A command-line program to download videos from youtube.com and many other [video platforms](docs/supportedsites.md)
 
@@ -12,8 +14,8 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
 
 * [NEW FEATURES](#new-features)
 * [INSTALLATION](#installation)
-    * [UPDATE](#update)
-    * [COMPILE](#compile)
+    * [Update](#update)
+    * [Compile](#compile)
 * [DESCRIPTION](#description)
 * [OPTIONS](#options)
     * [Network Options](#network-options)
@@ -30,7 +32,7 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
     * [Authentication Options](#authentication-options)
     * [Adobe Pass Options](#adobe-pass-options)
     * [Post-processing Options](#post-processing-options)
-    * [SponSkrub Options (SponsorBlock)](#sponSkrub-options-sponsorblock)
+    * [SponSkrub (SponsorBlock) Options](#sponskrub-sponsorblock-options)
     * [Extractor Options](#extractor-options)
 * [CONFIGURATION](#configuration)
     * [Authentication with .netrc file](#authentication-with-netrc-file)
@@ -41,63 +43,74 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
     * [Filtering Formats](#filtering-formats)
     * [Sorting Formats](#sorting-formats)
     * [Format Selection examples](#format-selection-examples)
-* [VIDEO SELECTION](#video-selection-1)
+* [PLUGINS](#plugins)
 * [MORE](#more)
 
 
 # NEW FEATURES
 The major new features from the latest release of [blackjack4494/yt-dlc](https://github.com/blackjack4494/yt-dlc) are:
 
-* **[SponSkrub Integration](#sponSkrub-options-sponsorblock)**: You can use [SponSkrub](https://github.com/faissaloo/SponSkrub) to mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
+* **[SponSkrub Integration](#sponskrub-sponsorblock-options)**: You can use [SponSkrub](https://github.com/pukkandan/SponSkrub) to mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
 
-* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now prefered instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
+* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* Merged with youtube-dl **v2020.01.08**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494)
+* **Merged with youtube-dl v2021.01.24.1**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+
+* **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/pukkandan/yt-dlp/pull/31) for details.
 
 * **Youtube improvements**:
     * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and support downloading multiple pages of content
     * Youtube search works correctly (`ytsearch:`, `ytsearchdate:`) along with Search URLs
     * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
 
-* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius
 
 * **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina
 
-* **New options**: `--list-formats-as-table`, `--write-link`, `--force-download-archive` etc
+* **Plugin support**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
 
-and many other features and patches. See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlc/commits) for the full list of changes
+* **Multiple paths**: You can give different paths for different types of files. You can also set a temporary path where intermediary files are downloaded to. See [`--paths`](https://github.com/pukkandan/yt-dlp/#:~:text=-P,%20--paths%20TYPE:PATH) for details
+
+<!-- Relative link doesn't work for "#:~:text=" -->
+
+* **Portable Configuration**: Configuration files are automatically loaded from the home and root directories. See [configuration](#configuration) for details
+
+* **Other new options**: `--parse-metadata`, `--list-formats-as-table`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
+
+* **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, `%(duration_string)s` in `-o`, faster archive checking, more [format selection options](#format-selection) etc
 
 
-**PS**: Some of these changes are already in youtube-dlc, but are still unreleased. See [this](changelog.md#unreleased-changes-in-blackjack4494yt-dlc) for details
+See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlp/commits) for the full list of changes
+
+
+**PS**: Some of these changes are already in youtube-dlc, but are still unreleased. See [this](Changelog.md#unreleased-changes-in-blackjack4494yt-dlc) for details
 
 If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the amount of changes are very large. Compare [options](#options) and [supported sites](docs/supportedsites.md) with youtube-dl's to get an idea of the massive number of features/patches [youtube-dlc](https://github.com/blackjack4494/yt-dlc) has accumulated.
 
 
 # INSTALLATION
 
-To use the latest version, simply download and run the [latest release](https://github.com/pukkandan/yt-dlc/releases/latest).
-Currently, there is no support for any package managers.
-
-If you want to install the current master branch
-
-    python -m pip install git+https://github.com/pukkandan/yt-dlc
+You can install yt-dlp using one of the following methods:
+* Use [PyPI package](https://pypi.org/project/yt-dlp): `python -m pip install --upgrade yt-dlp`
+* Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest)
+* Use pip+git: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp.git@release`
+* Install master branch: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp`
 
 ### UPDATE
-**DO NOT UPDATE using `-U` !** instead download binaries again
+`-U` does not work. Simply repeat the install process to update.
 
 ### COMPILE
 
 **For Windows**:
-To build the Windows executable yourself (without version info!)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen for embedding thumbnail in opus/ogg files)
+
+    python -m pip install --upgrade pyinstaller mutagen
+
+For the 64bit version, run `py devscripts\pyinst.py 64` using 64bit python3. Similarly, to install 32bit version, run `py devscripts\pyinst.py 32` using 32bit python (preferably 3)
+
+You can also build the executable without any version info or metadata by using:
 
-    python -m pip install --upgrade pyinstaller
     pyinstaller.exe youtube_dlc\__main__.py --onefile --name youtube-dlc
-    
-Or simply execute the `make_win.bat` if pyinstaller is installed.
-There will be a `youtube-dlc.exe` in `/dist`  
-
-New way to build Windows is to use `python pyinst.py` (please use python3 64Bit)  
-For 32Bit Version use a 32Bit Version of python (3 preferred here as well) and run `python pyinst32.py`  
 
 **For Unix**:
 You will need the required build tools  
@@ -106,6 +119,7 @@ Then simply type this
 
     make
 
+**Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number
 
 # DESCRIPTION
 **youtube-dlc** is a command-line program to download videos from youtube.com many other [video platforms](docs/supportedsites.md). It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
@@ -125,9 +139,9 @@ Then simply type this
                                      permissions (run with sudo if needed)
     -i, --ignore-errors              Continue on download errors, for example to
                                      skip unavailable videos in a playlist
-                                     (default) (Same as --no-abort-on-error)
-    --abort-on-error                 Abort downloading of further videos if an 
-                                     error occurs (Same as --no-ignore-errors)
+                                     (default) (Alias: --no-abort-on-error)
+    --abort-on-error                 Abort downloading of further videos if an
+                                     error occurs (Alias: --no-ignore-errors)
     --dump-user-agent                Display the current browser identification
     --list-extractors                List all supported extractors
     --extractor-descriptions         Output descriptions of all supported
@@ -142,23 +156,25 @@ Then simply type this
                                      warning when guessing). "error" just throws
                                      an error. The default value "fixup_error"
                                      repairs broken URLs, but emits an error if
-                                     this is not possible instead of searching.
-    --ignore-config, --no-config     Do not read configuration files. When given
-                                     in the global configuration file
-                                     /etc/youtube-dl.conf: Do not read the user
-                                     configuration in ~/.config/youtube-
-                                     dl/config (%APPDATA%/youtube-dl/config.txt
-                                     on Windows)
-    --config-location PATH           Location of the configuration file; either
-                                     the path to the config or its containing
-                                     directory.
+                                     this is not possible instead of searching
+    --ignore-config, --no-config     Disable loading any configuration files
+                                     except the one provided by --config-location.
+                                     When given inside a configuration
+                                     file, no further configuration files are
+                                     loaded. Additionally, (for backward
+                                     compatibility) if this option is found
+                                     inside the system configuration file, the
+                                     user configuration is not loaded
+    --config-location PATH           Location of the main configuration file;
+                                     either the path to the config or its
+                                     containing directory
     --flat-playlist                  Do not extract the videos of a playlist,
-                                     only list them.
+                                     only list them
     --flat-videos                    Do not resolve the video urls
     --no-flat-playlist               Extract the videos of a playlist
     --mark-watched                   Mark videos watched (YouTube only)
     --no-mark-watched                Do not mark videos watched
-    --no-color                       Do not emit color codes in output
+    --no-colors                      Do not emit color codes in output
 
 ## Network Options:
     --proxy URL                      Use the specified HTTP/HTTPS/SOCKS proxy.
@@ -176,7 +192,7 @@ Then simply type this
                                      some geo-restricted sites. The default
                                      proxy specified by --proxy (or none, if the
                                      option is not present) is used for the
-                                     actual downloading.
+                                     actual downloading
     --geo-bypass                     Bypass geographic restriction via faking
                                      X-Forwarded-For HTTP header
     --no-geo-bypass                  Do not bypass geographic restriction via
@@ -198,7 +214,7 @@ Then simply type this
                                      indexed 1, 2, 5, 8 in the playlist. You can
                                      specify range: "--playlist-items
                                      1-3,7,10-13", it will download the videos
-                                     at index 1, 2, 3, 7, 10, 11, 12 and 13.
+                                     at index 1, 2, 3, 7, 10, 11, 12 and 13
     --match-title REGEX              Download only matching titles (regex or
                                      caseless sub-string)
     --reject-title REGEX             Skip download for matching titles (regex or
@@ -222,36 +238,38 @@ Then simply type this
     --max-views COUNT                Do not download any videos with more than
                                      COUNT views
     --match-filter FILTER            Generic video filter. Specify any key (see
-                                     the "OUTPUT TEMPLATE" for a list of
-                                     available keys) to match if the key is
-                                     present, !key to check if the key is not
-                                     present, key > NUMBER (like "comment_count
-                                     > 12", also works with >=, <, <=, !=, =) to
-                                     compare against a number, key = 'LITERAL'
-                                     (like "uploader = 'Mike Smith'", also works
-                                     with !=) to match against a string literal
-                                     and & to require multiple matches. Values
-                                     which are not known are excluded unless you
-                                     put a question mark (?) after the operator.
-                                     For example, to only match videos that have
+                                     "OUTPUT TEMPLATE" for a list of available
+                                     keys) to match if the key is present, !key
+                                     to check if the key is not present,
+                                     key>NUMBER (like "comment_count > 12", also
+                                     works with >=, <, <=, !=, =) to compare
+                                     against a number, key = 'LITERAL' (like
+                                     "uploader = 'Mike Smith'", also works with
+                                     !=) to match against a string literal and &
+                                     to require multiple matches. Values which
+                                     are not known are excluded unless you put a
+                                     question mark (?) after the operator. For
+                                     example, to only match videos that have
                                      been liked more than 100 times and disliked
                                      less than 50 times (or the dislike
                                      functionality is not available at the given
                                      service), but who also have a description,
                                      use --match-filter "like_count > 100 &
-                                     dislike_count <? 50 & description" .
+                                     dislike_count <? 50 & description"
     --no-match-filter                Do not use generic video filter (default)
     --no-playlist                    Download only the video, if the URL refers
-                                     to a video and a playlist.
+                                     to a video and a playlist
     --yes-playlist                   Download the playlist, if the URL refers to
-                                     a video and a playlist.
+                                     a video and a playlist
     --age-limit YEARS                Download only videos suitable for the given
                                      age
     --download-archive FILE          Download only videos not listed in the
                                      archive file. Record the IDs of all
-                                     downloaded videos in it.
-    --break-on-existing              Stop the download process after attempting
-                                     to download a file that's in the archive.
+                                     downloaded videos in it
+    --break-on-existing              Stop the download process when encountering
+                                     a file that is in the archive
+    --break-on-reject                Stop the download process when encountering
+                                     a file that has been filtered out
     --no-download-archive            Do not use archive file (default)
     --include-ads                    Download advertisements as well
                                      (experimental)
@@ -261,15 +279,15 @@ Then simply type this
     -r, --limit-rate RATE            Maximum download rate in bytes per second
                                      (e.g. 50K or 4.2M)
     -R, --retries RETRIES            Number of retries (default is 10), or
-                                     "infinite".
+                                     "infinite"
     --fragment-retries RETRIES       Number of retries for a fragment (default
                                      is 10), or "infinite" (DASH, hlsnative and
                                      ISM)
     --skip-unavailable-fragments     Skip unavailable fragments for DASH,
                                      hlsnative and ISM (default)
-                                     (Same as --no-abort-on-unavailable-fragment)
+                                     (Alias: --no-abort-on-unavailable-fragment)
     --abort-on-unavailable-fragment  Abort downloading if a fragment is unavailable
-                                     (Same as --no-skip-unavailable-fragments)
+                                     (Alias: --no-skip-unavailable-fragments)
     --keep-fragments                 Keep downloaded fragments on disk after
                                      downloading is finished
     --no-keep-fragments              Delete downloaded fragments after
@@ -299,19 +317,36 @@ Then simply type this
                                      allowing to play the video while
                                      downloading (some players may not be able
                                      to play it)
-    --external-downloader COMMAND    Use the specified external downloader.
-                                     Currently supports
-                                     aria2c,avconv,axel,curl,ffmpeg,httpie,wget
-    --external-downloader-args ARGS  Give these arguments to the external
-                                     downloader
+    --external-downloader NAME       Use the specified external downloader.
+                                     Currently supports aria2c, avconv, axel,
+                                     curl, ffmpeg, httpie, wget
+    --downloader-args NAME:ARGS      Give these arguments to the external
+                                     downloader. Specify the downloader name and
+                                     the arguments separated by a colon ":". You
+                                     can use this option multiple times
+                                     (Alias: --external-downloader-args)
 
 ## Filesystem Options:
     -a, --batch-file FILE            File containing URLs to download ('-' for
                                      stdin), one URL per line. Lines starting
                                      with '#', ';' or ']' are considered as
-                                     comments and ignored.
-    -o, --output TEMPLATE            Output filename template, see the "OUTPUT
+                                     comments and ignored
+    -P, --paths TYPE:PATH            The paths where the files should be
+                                     downloaded. Specify the type of file and
+                                     the path separated by a colon ":"
+                                     (supported: description|annotation|subtitle
+                                     |infojson|thumbnail). Additionally, you can
+                                     also provide "home" and "temp" paths. All
+                                     intermediary files are first downloaded to
+                                     the temp path and then the final files are
+                                     moved over to the home path after download
+                                     is finished. Note that this option is
+                                     ignored if --output is an absolute path
+    -o, --output TEMPLATE            Output filename template, see "OUTPUT
                                      TEMPLATE" for details
+    --output-na-placeholder TEXT     Placeholder value for unavailable meta
+                                     fields in output filename template
+                                     (default: "NA")
     --autonumber-start NUMBER        Specify the start value for %(autonumber)s
                                      (default is 1)
     --restrict-filenames             Restrict filenames to only ASCII
@@ -343,6 +378,8 @@ Then simply type this
     --write-annotations              Write video annotations to a
                                      .annotations.xml file
     --no-write-annotations           Do not write video annotations (default)
+    --get-comments                   Retrieve video comments to be placed in the
+                                     .info.json file
     --load-info-json FILE            JSON file containing the video information
                                      (created with the "--write-info-json"
                                      option)
@@ -356,7 +393,7 @@ Then simply type this
                                      ~/.cache/youtube-dl . At the moment, only
                                      YouTube player files (for videos with
                                      obfuscated signatures) are cached, but that
-                                     may change.
+                                     may change
     --no-cache-dir                   Disable filesystem caching
     --rm-cache-dir                   Delete all filesystem cache files
     --trim-file-name LENGTH          Limit the filename length (extension
@@ -371,13 +408,13 @@ Then simply type this
                                      formats
 
 ## Internet Shortcut Options:
-    --write-link                     Write an internet shortcut file, depending on 
-                                     the current platform (.url/.webloc/.desktop). 
-                                     The URL may be cached by the OS.
-    --write-url-link                 Write a Windows .url internet shortcut file. 
-                                     (The OS caches the URL based on the file path)
-    --write-webloc-link              Write a .webloc macOS internet shortcut file
-    --write-desktop-link             Write a .desktop Linux internet shortcut file
+    --write-link                     Write an internet shortcut file, depending
+                                     on the current platform (.url, .webloc or
+                                     .desktop). The URL may be cached by the OS
+    --write-url-link                 Write a .url Windows internet shortcut. The
+                                     OS caches the URL based on the file path
+    --write-webloc-link              Write a .webloc macOS internet shortcut
+    --write-desktop-link             Write a .desktop Linux internet shortcut
 
 ## Verbosity / Simulation Options:
     -q, --quiet                      Activate quiet mode
@@ -394,18 +431,18 @@ Then simply type this
     --get-filename                   Simulate, quiet but print output filename
     --get-format                     Simulate, quiet but print output format
     -j, --dump-json                  Simulate, quiet but print JSON information.
-                                     See the "OUTPUT TEMPLATE" for a description
-                                     of available keys.
+                                     See "OUTPUT TEMPLATE" for a description of
+                                     available keys
     -J, --dump-single-json           Simulate, quiet but print JSON information
                                      for each command-line argument. If the URL
                                      refers to a playlist, dump the whole
-                                     playlist information in a single line.
+                                     playlist information in a single line
     --print-json                     Be quiet and print the video information as
-                                     JSON (video is still being downloaded).
-    --force-write-archive            Force download archive entries to be written 
-                                     as far as no errors occur, even if -s or 
-                                     another simulation switch is used.
-                                     (Same as --force-download-archive)
+                                     JSON (video is still being downloaded)
+    --force-write-archive            Force download archive entries to be
+                                     written as far as no errors occur,even if
+                                     -s or another simulation switch is used
+                                     (Alias: --force-download-archive)
     --newline                        Output progress bar as new lines
     --no-progress                    Do not print progress bar
     --console-title                  Display progress in console titlebar
@@ -431,7 +468,7 @@ Then simply type this
     --referer URL                    Specify a custom referer, use if the video
                                      access is restricted to one domain
     --add-header FIELD:VALUE         Specify a custom HTTP header and its value,
-                                     separated by a colon ':'. You can use this
+                                     separated by a colon ":". You can use this
                                      option multiple times
     --bidi-workaround                Work around terminals that lack
                                      bidirectional text support. Requires bidiv
@@ -441,11 +478,11 @@ Then simply type this
                                      of a range for randomized sleep before each
                                      download (minimum possible number of
                                      seconds to sleep) when used along with
-                                     --max-sleep-interval.
+                                     --max-sleep-interval
     --max-sleep-interval SECONDS     Upper bound of a range for randomized sleep
                                      before each download (maximum possible
                                      number of seconds to sleep). Must only be
-                                     used along with --min-sleep-interval.
+                                     used along with --min-sleep-interval
     --sleep-subtitles SECONDS        Enforce sleep interval on subtitles as well
 
 ## Video Format Options:
@@ -453,8 +490,8 @@ Then simply type this
                                      for more details
     -S, --format-sort SORTORDER      Sort the formats by the fields given, see
                                      "Sorting Formats" for more details
-    --S-force, --format-sort-force   Force user specified sort order to have 
-                                     precedence over all fields, see "Sorting 
+    --S-force, --format-sort-force   Force user specified sort order to have
+                                     precedence over all fields, see "Sorting
                                      Formats" for more details
     --no-format-sort-force           Some fields have precedence over the user
                                      specified sort order (default), see
@@ -472,22 +509,22 @@ Then simply type this
                                      formats of same quality
     -F, --list-formats               List all available formats of requested
                                      videos
-    --list-formats-as-table          Present the output of -F in a more tabular
-                                     form (default)
-                                     (Same as --no-list-formats-as-table)
+    --list-formats-as-table          Present the output of -F in tabular form
+                                     (default)
     --list-formats-old               Present the output of -F in the old form
-    --youtube-include-dash-manifest  Download the DASH manifests and related data 
-                                     on YouTube videos (default)
-                                     (Same as --no-youtube-skip-dash-manifest)
+                                     (Alias: --no-list-formats-as-table)
+    --youtube-include-dash-manifest  Download the DASH manifests and related
+                                     data on YouTube videos (default)
+                                     (Alias: --no-youtube-skip-dash-manifest)
     --youtube-skip-dash-manifest     Do not download the DASH manifests and
                                      related data on YouTube videos
-                                     (Same as --no-youtube-include-dash-manifest)
-    --youtube-include-hls-manifest   Download the HLS manifests and related data 
+                                     (Alias: --no-youtube-include-dash-manifest)
+    --youtube-include-hls-manifest   Download the HLS manifests and related data
                                      on YouTube videos (default)
-                                     (Same as --no-youtube-skip-hls-manifest)
+                                     (Alias: --no-youtube-skip-hls-manifest)
     --youtube-skip-hls-manifest      Do not download the HLS manifests and
                                      related data on YouTube videos
-                                     (Same as --no-youtube-include-hls-manifest)
+                                     (Alias: --no-youtube-include-hls-manifest)
     --merge-output-format FORMAT     If a merge is required (e.g.
                                      bestvideo+bestaudio), output to given
                                      container format. One of mkv, mp4, ogg,
@@ -513,7 +550,7 @@ Then simply type this
 ## Authentication Options:
     -u, --username USERNAME          Login with this account ID
     -p, --password PASSWORD          Account password. If this option is left
-                                     out, youtube-dlc will ask interactively.
+                                     out, youtube-dlc will ask interactively
     -2, --twofactor TWOFACTOR        Two-factor authentication code
     -n, --netrc                      Use .netrc authentication data
     --video-password PASSWORD        Video password (vimeo, youku)
@@ -525,41 +562,51 @@ Then simply type this
     --ap-username USERNAME           Multiple-system operator account login
     --ap-password PASSWORD           Multiple-system operator account password.
                                      If this option is left out, youtube-dlc
-                                     will ask interactively.
+                                     will ask interactively
     --ap-list-mso                    List all supported multiple-system
                                      operators
 
 ## Post-Processing Options:
     -x, --extract-audio              Convert video files to audio-only files
-                                     (requires ffmpeg or avconv and ffprobe or
-                                     avprobe)
+                                     (requires ffmpeg and ffprobe)
     --audio-format FORMAT            Specify audio format: "best", "aac",
                                      "flac", "mp3", "m4a", "opus", "vorbis", or
                                      "wav"; "best" by default; No effect without
                                      -x
-    --audio-quality QUALITY          Specify ffmpeg/avconv audio quality, insert
-                                     a value between 0 (better) and 9 (worse)
-                                     for VBR or a specific bitrate like 128K
+    --audio-quality QUALITY          Specify ffmpeg audio quality, insert a
+                                     value between 0 (better) and 9 (worse) for
+                                     VBR or a specific bitrate like 128K
                                      (default 5)
     --remux-video FORMAT             Remux the video into another container if
-                                     necessary (currently supported: mp4|mkv).
-                                     If target container does not support the
-                                     video/audio codec, remuxing will fail
+                                     necessary (currently supported: mp4|mkv|flv
+                                     |webm|mov|avi|mp3|mka|m4a|ogg|opus). If
+                                     target container does not support the
+                                     video/audio codec, remuxing will fail. You
+                                     can specify multiple rules; eg.
+                                     "aac>m4a/mov>mp4/mkv" will remux aac to
+                                     m4a, mov to mp4 and anything else to mkv.
     --recode-video FORMAT            Re-encode the video into another format if
-                                     re-encoding is necessary (currently
-                                     supported: mp4|flv|ogg|webm|mkv|avi)
+                                     re-encoding is necessary. The supported
+                                     formats are the same as --remux-video
     --postprocessor-args NAME:ARGS   Give these arguments to the postprocessors.
-                                     Specify the postprocessor name and the
-                                     arguments separated by a colon ':' to give
-                                     the argument to only the specified
-                                     postprocessor. Supported names are
+                                     Specify the postprocessor/executable name
+                                     and the arguments separated by a colon ":"
+                                     to give the argument to the specified
+                                     postprocessor/executable. Supported
+                                     postprocessors are: SponSkrub,
                                      ExtractAudio, VideoRemuxer, VideoConvertor,
                                      EmbedSubtitle, Metadata, Merger,
                                      FixupStretched, FixupM4a, FixupM3u8,
-                                     SubtitlesConvertor, SponSkrub and Default.
-                                     You can use this option multiple times to
-                                     give different arguments to different
-                                     postprocessors
+                                     SubtitlesConvertor and EmbedThumbnail. The
+                                     supported executables are: SponSkrub,
+                                     FFmpeg, FFprobe, and AtomicParsley. You can
+                                     use this option multiple times to give
+                                     different arguments to different
+                                     postprocessors. You can also specify
+                                     "PP+EXE:ARGS" to give the arguments to the
+                                     specified executable only when being used
+                                     by the specified postprocessor. You can use
+                                     this option multiple times (Alias: --ppa)
     -k, --keep-video                 Keep the intermediate video file on disk
                                      after post-processing
     --no-keep-video                  Delete the intermediate video file after
@@ -573,16 +620,20 @@ Then simply type this
     --no-embed-thumbnail             Do not embed thumbnail (default)
     --add-metadata                   Write metadata to the video file
     --no-add-metadata                Do not write metadata (default)
-    --metadata-from-title FORMAT     Parse additional metadata like song title /
-                                     artist from the video title. The format
-                                     syntax is the same as --output. Regular
-                                     expression with named capture groups may
+    --parse-metadata FIELD:FORMAT    Parse additional metadata like title/artist
+                                     from other fields. Give field name to
+                                     extract data from, and format of the field
+                                     seperated by a ":". Either regular
+                                     expression with named capture groups or a
+                                     similar syntax to the output template can
                                      also be used. The parsed parameters replace
-                                     existing values. Example: --metadata-from-
-                                     title "%(artist)s - %(title)s" matches a
+                                     any existing values and can be use in
+                                     output templateThis option can be used
+                                     multiple times. Example: --parse-metadata
+                                     "title:%(artist)s - %(title)s" matches a
                                      title like "Coldplay - Paradise". Example
-                                     (regex): --metadata-from-title
-                                     "(?P<artist>.+?) - (?P<title>.+)"
+                                     (regex): --parse-metadata
+                                     "description:Artist - (?P<artist>.+?)"
     --xattrs                         Write metadata to the video file's xattrs
                                      (using dublin core and xdg standards)
     --fixup POLICY                   Automatically correct known faults of the
@@ -590,14 +641,9 @@ Then simply type this
                                      emit a warning), detect_or_warn (the
                                      default; fix file if we can, warn
                                      otherwise)
-    --prefer-avconv                  Prefer avconv over ffmpeg for running the
-                                     postprocessors (Same as --no-prefer-ffmpeg)
-    --prefer-ffmpeg                  Prefer ffmpeg over avconv for running the
-                                     postprocessors (default)
-                                     (Same as --no-prefer-avconv)
-    --ffmpeg-location PATH           Location of the ffmpeg/avconv binary;
-                                     either the path to the binary or its
-                                     containing directory.
+    --ffmpeg-location PATH           Location of the ffmpeg binary; either the
+                                     path to the binary or its containing
+                                     directory
     --exec CMD                       Execute a command on the file after
                                      downloading and post-processing, similar to
                                      find's -exec syntax. Example: --exec 'adb
@@ -605,11 +651,14 @@ Then simply type this
     --convert-subs FORMAT            Convert the subtitles to other format
                                      (currently supported: srt|ass|vtt|lrc)
 
-## [SponSkrub](https://github.com/faissaloo/SponSkrub) Options ([SponsorBlock](https://sponsor.ajay.app)):
-    --sponskrub                      Use sponskrub to mark sponsored sections
-                                     with the data available in SponsorBlock
-                                     API. This is enabled by default if the
-                                     sponskrub binary exists (Youtube only)
+## SponSkrub (SponsorBlock) Options:
+[SponSkrub](https://github.com/pukkandan/SponSkrub) is a utility to
+    mark/remove sponsor segments from downloaded YouTube videos using
+    [SponsorBlock API](https://sponsor.ajay.app)
+
+    --sponskrub                      Use sponskrub to mark sponsored sections.
+                                     This is enabled by default if the sponskrub
+                                     binary exists (Youtube only)
     --no-sponskrub                   Do not use sponskrub
     --sponskrub-cut                  Cut out the sponsor sections instead of
                                      simply marking them
@@ -621,19 +670,33 @@ Then simply type this
                                      video was already downloaded (default)
     --sponskrub-location PATH        Location of the sponskrub binary; either
                                      the path to the binary or its containing
-                                     directory.
+                                     directory
 
 ## Extractor Options:
-    --ignore-dynamic-mpd             Do not process dynamic DASH manifests
-                                     (Same as --no-allow-dynamic-mpd)
     --allow-dynamic-mpd              Process dynamic DASH manifests (default)
-                                     (Same as --no-ignore-dynamic-mpd)
+                                     (Alias: --no-ignore-dynamic-mpd)
+    --ignore-dynamic-mpd             Do not process dynamic DASH manifests
+                                     (Alias: --no-allow-dynamic-mpd)
 
 # CONFIGURATION
 
-You can configure youtube-dlc by placing any supported command line option to a configuration file. On Linux and macOS, the system wide configuration file is located at `/etc/youtube-dlc.conf` and the user wide configuration file at `~/.config/youtube-dlc/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dlc\config.txt` or `C:\Users\<user name>\youtube-dlc.conf`. Note that by default configuration file may not exist so you may need to create it yourself.
+You can configure youtube-dlc by placing any supported command line option to a configuration file. The configuration is loaded from the following locations:
 
-For example, with the following configuration file youtube-dlc will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory:
+1. **Main Configuration**: The file given by `--config-location`
+1. **Portable Configuration**: `yt-dlp.conf` or `youtube-dlc.conf` in the same directory as the bundled binary. If you are running from source-code (`<root dir>/youtube_dlc/__main__.py`), the root directory is used instead.
+1. **Home Configuration**: `yt-dlp.conf` or `youtube-dlc.conf` in the home path given by `-P "home:<path>"`, or in the current directory if no such path is given
+1. **User Configuration**:
+    * `%XDG_CONFIG_HOME%/yt-dlp/config` (recommended on Linux/macOS)
+    * `%XDG_CONFIG_HOME%/yt-dlp.conf`
+    * `%APPDATA%/yt-dlp/config` (recommended on Windows)
+    * `%APPDATA%/yt-dlp/config.txt`
+    * `~/yt-dlp.conf`
+    * `~/yt-dlp.conf.txt`
+
+    If none of these files are found, the search is performed again by replacing `yt-dlp` with `youtube-dlc`. Note that `~` points to `C:\Users\<user name>` on windows. Also, `%XDG_CONFIG_HOME%` defaults to `~/.config` if undefined
+1. **System Configuration**: `/etc/yt-dlp.conf` or `/etc/youtube-dlc.conf`
+
+For example, with the following configuration file youtube-dlc will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory:
 ```
 # Lines starting with # are comments
 
@@ -646,15 +709,13 @@ For example, with the following configuration file youtube-dlc will always extra
 # Use this proxy
 --proxy 127.0.0.1:3128
 
-# Save all videos under Movies directory in your home directory
--o ~/Movies/%(title)s.%(ext)s
+# Save all videos under YouTube directory in your home directory
+-o ~/YouTube/%(title)s.%(ext)s
 ```
 
-Note that options in configuration file are just the same options aka switches used in regular command line calls thus there **must be no whitespace** after `-` or `--`, e.g. `-o` or `--proxy` but not `- o` or `-- proxy`.
+Note that options in configuration file are just the same options aka switches used in regular command line calls; thus there **must be no whitespace** after `-` or `--`, e.g. `-o` or `--proxy` but not `- o` or `-- proxy`.
 
-You can use `--ignore-config` if you want to disable the configuration file for a particular youtube-dlc run.
-
-You can also use `--config-location` if you want to use custom configuration file for a particular youtube-dlc run.
+You can use `--ignore-config` if you want to disable all configuration files for a particular youtube-dlc run. If `--ignore-config` is found inside any configuration file, no further configuration will be loaded. For example, having the option in the portable configuration file prevents loading of user and system configurations. Additionally, (for backward compatibility) if `--ignore-config` is found inside the system configuration file, the user configuration is not loaded.
 
 ### Authentication with `.netrc` file
 
@@ -681,7 +742,7 @@ set HOME=%USERPROFILE%
 
 # OUTPUT TEMPLATE
 
-The `-o` option allows users to indicate a template for the output file names.
+The `-o` option is used to indicate a template for the output file names while `-P` option is used to specify the path each type of file should be saved to.
 
 **tl;dr:** [navigate me to examples](#output-template-examples).
 
@@ -715,7 +776,7 @@ The basic usage is not to set any template arguments when downloading a single f
  - `is_live` (boolean): Whether this video is a live stream or a fixed-length video
  - `start_time` (numeric): Time in seconds where the reproduction should start, as specified in the URL
  - `end_time` (numeric): Time in seconds where the reproduction should end, as specified in the URL
- - `format` (string): A human-readable description of the format 
+ - `format` (string): A human-readable description of the format
  - `format_id` (string): Format code specified by `--format`
  - `format_note` (string): Additional info about the format
  - `width` (numeric): Width of the video
@@ -772,7 +833,7 @@ Available for the media that is a track or a part of a music album:
  - `disc_number` (numeric): Number of the disc or other physical medium the track belongs to
  - `release_year` (numeric): Year (YYYY) when the album was released
 
-Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with `NA`.
+Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
 
 For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `youtube-dlc test video` and id `BaW_jenozKcj`, this will result in a `youtube-dlc test video-BaW_jenozKcj.mp4` file created in the current directory.
 
@@ -820,7 +881,7 @@ $ youtube-dlc -o - BaW_jenozKc
 # FORMAT SELECTION
 
 By default, youtube-dlc tries to download the best available quality if you **don't** pass any options.
-This is generally equivalent to using `-f bestvideo*+bestaudio/best`. However, if multiple audiostreams is enabled (`--audio-multistreams`), the default format changes to `-f bestvideo+bestaudio/best`. Similarly, if ffmpeg and avconv are unavailable, or if you use youtube-dlc to stream to `stdout` (`-o -`), the default becomes `-f best/bestvideo+bestaudio`.
+This is generally equivalent to using `-f bestvideo*+bestaudio/best`. However, if multiple audiostreams is enabled (`--audio-multistreams`), the default format changes to `-f bestvideo+bestaudio/best`. Similarly, if ffmpeg is unavailable, or if you use youtube-dlc to stream to `stdout` (`-o -`), the default becomes `-f best/bestvideo+bestaudio`.
 
 The general syntax for format selection is `--f FORMAT` (or `--format FORMAT`) where `FORMAT` is a *selector expression*, i.e. an expression that describes format or formats you would like to download.
 
@@ -851,7 +912,7 @@ If you want to download multiple videos and they don't have the same formats ava
 
 If you want to download several formats of the same video use a comma as a separator, e.g. `-f 22,17,18` will download all these three formats, of course if they are available. Or a more sophisticated example combined with the precedence feature: `-f 136/137/mp4/bestvideo,140/m4a/bestaudio`.
 
-You can merge the video and audio of multiple formats into a single file using `-f <format1>+<format2>+...` (requires ffmpeg or avconv installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg/avconv. If `--no-video-multistreams` is used, all formats with a video stream except the first one are ignored. Similarly, if `--no-audio-multistreams` is used, all formats with an audio stream except the first one are ignored. For example, `-f bestvideo+best+bestaudio` will download and merge all 3 given formats. The resulting file will have 2 video streams and 2 audio streams. But `-f bestvideo+best+bestaudio --no-video-multistreams` will download and merge only `bestvideo` and `bestaudio`. `best` is ignored since another format containing a video stream (`bestvideo`) has already been selected. The order of the formats is therefore important. `-f best+bestaudio --no-audio-multistreams` will download and merge both formats while `-f bestaudio+best --no-audio-multistreams` will ignore `best` and download only `bestaudio`.
+You can merge the video and audio of multiple formats into a single file using `-f <format1>+<format2>+...` (requires ffmpeg installed), for example `-f bestvideo+bestaudio` will download the best video-only format, the best audio-only format and mux them together with ffmpeg. If `--no-video-multistreams` is used, all formats with a video stream except the first one are ignored. Similarly, if `--no-audio-multistreams` is used, all formats with an audio stream except the first one are ignored. For example, `-f bestvideo+best+bestaudio` will download and merge all 3 given formats. The resulting file will have 2 video streams and 2 audio streams. But `-f bestvideo+best+bestaudio --no-video-multistreams` will download and merge only `bestvideo` and `bestaudio`. `best` is ignored since another format containing a video stream (`bestvideo`) has already been selected. The order of the formats is therefore important. `-f best+bestaudio --no-audio-multistreams` will download and merge both formats while `-f bestaudio+best --no-audio-multistreams` will ignore `best` and download only `bestaudio`.
 
 ## Filtering Formats
 
@@ -890,35 +951,35 @@ Format selectors can also be grouped using parentheses, for example if you want
 
 You can change the criteria for being considered the `best` by using `-S` (`--format-sort`). The general format for this is `--format-sort field1,field2...`. The available fields are:
 
- - `video`, `has_video`: Gives priority to formats that has a video stream
- - `audio`, `has_audio`: Gives priority to formats that has a audio stream
- - `extractor`, `preference`, `extractor_preference`: The format preference as given by the extractor
- - `lang`, `language_preference`: Language preference as given by the extractor
+ - `hasvid`: Gives priority to formats that has a video stream
+ - `hasaud`: Gives priority to formats that has a audio stream
+ - `ie_pref`: The format preference as given by the extractor
+ - `lang`: Language preference as given by the extractor
  - `quality`: The quality of the format. This is a metadata field available in some websites
- - `source`, `source_preference`: Preference of the source as given by the extractor
- - `proto`, `protocol`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- - `vcodec`, `video_codec`: Video Codec (`vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
- - `acodec`, `audio_codec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
+ - `source`: Preference of the source as given by the extractor
+ - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
+ - `vcodec`: Video Codec (`vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
+ - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
  - `codec`: Equivalent to `vcodec,acodec`
- - `vext`, `video_ext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
- - `aext`, `audio_ext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other > unknown). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`.
- - `ext`, `extension`: Equivalent to `vext,aext`
+ - `vext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
+ - `aext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other > unknown). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`.
+ - `ext`: Equivalent to `vext,aext`
  - `filesize`: Exact filesize, if know in advance. This will be unavailable for mu38 and DASH formats.
- - `filesize_approx`: Approximate filesize calculated from the manifests
- - `size`, `filesize_estimate`: Exact filesize if available, otherwise approximate filesize
+ - `fs_approx`: Approximate filesize calculated from the manifests
+ - `size`: Exact filesize if available, otherwise approximate filesize
  - `height`: Height of video
  - `width`: Width of video
- - `res`, `dimension`: Video resolution, calculated as the smallest dimension.
- - `fps`, `framerate`: Framerate of video
- - `tbr`, `total_bitrate`: Total average bitrate in KBit/s
- - `vbr`, `video_bitrate`: Average video bitrate in KBit/s
- - `abr`, `audio_bitrate`: Average audio bitrate in KBit/s
- - `br`, `bitrate`: Equivalent to using `tbr,vbr,abr`
- - `samplerate`, `asr`: Audio sample rate in Hz
+ - `res`: Video resolution, calculated as the smallest dimension.
+ - `fps`: Framerate of video
+ - `tbr`: Total average bitrate in KBit/s
+ - `vbr`: Average video bitrate in KBit/s
+ - `abr`: Average audio bitrate in KBit/s
+ - `br`: Equivalent to using `tbr,vbr,abr`
+ - `asr`: Audio sample rate in Hz
 
 Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
-The fields `has_video`, `extractor`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec,size,br,asr,proto,ext,has_audio,source,format_id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
 
 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 
@@ -957,7 +1018,7 @@ $ youtube-dlc -f 'wv*+wa/w'
 $ youtube-dlc -S '+res'
 
 # Download the smallest video available
-$ youtube-dlc -S '+size,+bitrate'
+$ youtube-dlc -S '+size,+br'
 
 
 
@@ -1005,7 +1066,7 @@ $ youtube-dlc -f '(bv*+ba/b)[protocol^=http][protocol!*=dash] / (bv*+ba/b)'
 
 # Download best video available via the best protocol
 # (https/ftps > http/ftp > m3u8_native > m3u8 > http_dash_segments ...)
-$ youtube-dlc -S 'protocol'
+$ youtube-dlc -S 'proto'
 
 
 
@@ -1041,9 +1102,11 @@ $ youtube-dlc -S 'res:720,fps'
 $ youtube-dlc -S '+res:480,codec,br'
 ```
 
+# PLUGINS
 
+Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`. Currently only `extractor` plugins are supported. Support for `downloader` and `postprocessor` plugins may be added in the future. See [ytdlp_plugins](ytdlp_plugins) for example.
 
-
+**Note**: `<root-dir>` is the directory of the binary (`<root-dir>/youtube-dlc`), or the root directory of the module if you are running directly from source-code (`<root dir>/youtube_dlc/__main__.py`)
 
 # MORE
-For FAQ, Developer Instructions etc., see the [original README](https://github.com/ytdl-org/youtube-dl)
+For FAQ, Developer Instructions etc., see the [original README](https://github.com/ytdl-org/youtube-dl#faq)

devscripts/pyinst.py

@@ -0,0 +1,69 @@
+from __future__ import unicode_literals
+import sys
+
+from PyInstaller.utils.win32.versioninfo import (
+    VarStruct, VarFileInfo, StringStruct, StringTable,
+    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
+)
+import PyInstaller.__main__
+
+
+assert len(sys.argv) > 1 and sys.argv[1] in ("32", "64")
+_x86 = "_x86" if sys.argv[1] == "32" else ""
+
+FILE_DESCRIPTION = 'Media Downloader%s' % (" (32 Bit)" if _x86 else '')
+SHORT_URLS = {"32": "git.io/JUGsM", "64": "git.io/JLh7K"}
+
+
+exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
+VERSION = locals()['__version__']
+
+VERSION_LIST = VERSION.replace('-', '.').split('.')
+VERSION_LIST = list(map(int, VERSION_LIST)) + [0] * (4 - len(VERSION_LIST))
+
+print('Version: %s%s' % (VERSION, _x86))
+print('Remember to update the version using devscipts\\update-version.py')
+
+VERSION_FILE = VSVersionInfo(
+    ffi=FixedFileInfo(
+        filevers=VERSION_LIST,
+        prodvers=VERSION_LIST,
+        mask=0x3F,
+        flags=0x0,
+        OS=0x4,
+        fileType=0x1,
+        subtype=0x0,
+        date=(0, 0),
+    ),
+    kids=[
+        StringFileInfo([
+            StringTable(
+                "040904B0", [
+                    StringStruct("Comments", "Youtube-dlc%s Command Line Interface." % _x86),
+                    StringStruct("CompanyName", "pukkandan@gmail.com"),
+                    StringStruct("FileDescription", FILE_DESCRIPTION),
+                    StringStruct("FileVersion", VERSION),
+                    StringStruct("InternalName", "youtube-dlc%s" % _x86),
+                    StringStruct(
+                        "LegalCopyright",
+                        "pukkandan@gmail.com | UNLICENSE",
+                    ),
+                    StringStruct("OriginalFilename", "youtube-dlc%s.exe" % _x86),
+                    StringStruct("ProductName", "Youtube-dlc%s" % _x86),
+                    StringStruct("ProductVersion", "%s%s | %s" % (VERSION, _x86, SHORT_URLS[sys.argv[1]])),
+                ])]),
+        VarFileInfo([VarStruct("Translation", [0, 1200])])
+    ]
+)
+
+PyInstaller.__main__.run([
+    '--name=youtube-dlc%s' % _x86,
+    '--onefile',
+    '--icon=devscripts/cloud.ico',
+    '--exclude-module=youtube_dl',
+    '--exclude-module=test',
+    '--exclude-module=ytdlp_plugins',
+    '--hidden-import=mutagen',
+    'youtube_dlc/__main__.py',
+])
+SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)

devscripts/update-version.py

@@ -0,0 +1,31 @@
+from __future__ import unicode_literals
+from datetime import datetime
+# import urllib.request
+
+# response = urllib.request.urlopen('https://blackjack4494.github.io/youtube-dlc/update/LATEST_VERSION')
+# old_version = response.read().decode('utf-8')
+
+exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
+old_version = locals()['__version__']
+
+old_version_list = old_version.replace('-', '.').split(".", 4)
+
+old_ver = '.'.join(old_version_list[:3])
+old_rev = old_version_list[3] if len(old_version_list) > 3 else ''
+
+ver = datetime.now().strftime("%Y.%m.%d")
+rev = str(int(old_rev or 0) + 1) if old_ver == ver else ''
+
+VERSION = '.'.join((ver, rev)) if rev else ver
+# VERSION_LIST = [(int(v) for v in ver.split(".") + [rev or 0])]
+
+print('::set-output name=ytdlc_version::' + VERSION)
+
+file_version_py = open('youtube_dlc/version.py', 'rt')
+data = file_version_py.read()
+data = data.replace(old_version, VERSION)
+file_version_py.close()
+
+file_version_py = open('youtube_dlc/version.py', 'wt')
+file_version_py.write(data)
+file_version_py.close()

docs/supportedsites.md

@@ -47,12 +47,13 @@
  - **Amara**
  - **AMCNetworks**
  - **AmericasTestKitchen**
+ - **AmericasTestKitchenSeason**
  - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **AnimeLab**
  - **AnimeLabShows**
  - **AnimeOnDemand**
  - **Anvato**
- - **aol.com**
+ - **aol.com**: Yahoo screen and movies
  - **APA**
  - **Aparat**
  - **AppleConnect**
@@ -79,6 +80,8 @@
  - **AudioBoom**
  - **audiomack**
  - **audiomack:album**
+ - **Audius**
+ - **audius:playlist**
  - **AWAAN**
  - **awaan:live**
  - **awaan:season**
@@ -111,7 +114,9 @@
  - **BiliBili**
  - **BilibiliAudio**
  - **BilibiliAudioAlbum**
+ - **BilibiliChannel**
  - **BiliBiliPlayer**
+ - **BiliBiliSearch**: Bilibili video search, "bilisearch" keyword
  - **BioBioChileTV**
  - **Biography**
  - **BIQLE**
@@ -197,8 +202,6 @@
  - **CNNArticle**
  - **CNNBlogs**
  - **ComedyCentral**
- - **ComedyCentralFullEpisodes**
- - **ComedyCentralShortname**
  - **ComedyCentralTV**
  - **CondeNast**: Condé Nast media group: Allure, Architectural Digest, Ars Technica, Bon Appétit, Brides, Condé Nast, Condé Nast Traveler, Details, Epicurious, GQ, Glamour, Golf Digest, SELF, Teen Vogue, The New Yorker, Vanity Fair, Vogue, W Magazine, WIRED
  - **CONtv**
@@ -431,7 +434,8 @@
  - **Katsomo**
  - **KeezMovies**
  - **Ketnet**
- - **KhanAcademy**
+ - **khanacademy**
+ - **khanacademy:unit**
  - **KickStarter**
  - **KinjaEmbed**
  - **KinoPoisk**
@@ -519,6 +523,12 @@
  - **Mgoon**
  - **MGTV**: 芒果TV
  - **MiaoPai**
+ - **mildom**: Record ongoing live by specific user in Mildom
+ - **mildom:user:vod**: Download all VODs from specific user in Mildom
+ - **mildom:vod**: Download a VOD in Mildom
+ - **minds**
+ - **minds:channel**
+ - **minds:group**
  - **MinistryGrid**
  - **Minoto**
  - **miomio.tv**
@@ -879,6 +889,8 @@
  - **Sport5**
  - **SportBox**
  - **SportDeutschland**
+ - **spotify**
+ - **spotify:show**
  - **Spreaker**
  - **SpreakerPage**
  - **SpreakerShow**
@@ -961,12 +973,13 @@
  - **TNAFlixNetworkEmbed**
  - **toggle**
  - **ToonGoggles**
- - **Tosh**: Tosh.0
  - **tou.tv**
  - **Toypics**: Toypics video
  - **ToypicsUser**: Toypics user profile
  - **TrailerAddict** (Currently broken)
  - **Trilulilu**
+ - **Trovo**
+ - **TrovoVod**
  - **TruNews**
  - **TruTV**
  - **Tube8**

make_win.bat

@@ -1 +0,0 @@
-py -m PyInstaller youtube_dlc\__main__.py --onefile --name youtube-dlc --version-file win\ver.txt --icon win\icon\cloud.ico --upx-exclude=vcruntime140.dll

pyinst.py

@@ -1,92 +0,0 @@
-from __future__ import unicode_literals
-from PyInstaller.utils.win32.versioninfo import (
-    VarStruct, VarFileInfo, StringStruct, StringTable,
-    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
-)
-import PyInstaller.__main__
-
-from datetime import datetime
-
-FILE_DESCRIPTION = 'Media Downloader'
-
-exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
-
-_LATEST_VERSION = locals()['__version__']
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))
-
-print(version)
-
-version_list = ver.split(".")
-_year, _month, _day = [int(value) for value in version_list]
-_rev = 0
-if rev:
-    _rev = rev
-_ver_tuple = _year, _month, _day, _rev
-
-version_file = VSVersionInfo(
-    ffi=FixedFileInfo(
-        filevers=_ver_tuple,
-        prodvers=_ver_tuple,
-        mask=0x3F,
-        flags=0x0,
-        OS=0x4,
-        fileType=0x1,
-        subtype=0x0,
-        date=(0, 0),
-    ),
-    kids=[
-        StringFileInfo(
-            [
-                StringTable(
-                    "040904B0",
-                    [
-                        StringStruct("Comments", "Youtube-dlc Command Line Interface."),
-                        StringStruct("CompanyName", "theidel@uni-bremen.de"),
-                        StringStruct("FileDescription", FILE_DESCRIPTION),
-                        StringStruct("FileVersion", version),
-                        StringStruct("InternalName", "youtube-dlc"),
-                        StringStruct(
-                            "LegalCopyright",
-                            "theidel@uni-bremen.de | UNLICENSE",
-                        ),
-                        StringStruct("OriginalFilename", "youtube-dlc.exe"),
-                        StringStruct("ProductName", "Youtube-dlc"),
-                        StringStruct("ProductVersion", version + " | git.io/JLh7K"),
-                    ],
-                )
-            ]
-        ),
-        VarFileInfo([VarStruct("Translation", [0, 1200])])
-    ]
-)
-
-PyInstaller.__main__.run([
-    '--name=youtube-dlc',
-    '--onefile',
-    '--icon=win/icon/cloud.ico',
-    'youtube_dlc/__main__.py',
-])
-SetVersion('dist/youtube-dlc.exe', version_file)

pyinst32.py

@@ -1,92 +0,0 @@
-from __future__ import unicode_literals
-from PyInstaller.utils.win32.versioninfo import (
-    VarStruct, VarFileInfo, StringStruct, StringTable,
-    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
-)
-import PyInstaller.__main__
-
-from datetime import datetime
-
-FILE_DESCRIPTION = 'Media Downloader 32 Bit Version'
-
-exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
-
-_LATEST_VERSION = locals()['__version__']
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))
-
-print(version)
-
-version_list = ver.split(".")
-_year, _month, _day = [int(value) for value in version_list]
-_rev = 0
-if rev:
-    _rev = rev
-_ver_tuple = _year, _month, _day, _rev
-
-version_file = VSVersionInfo(
-    ffi=FixedFileInfo(
-        filevers=_ver_tuple,
-        prodvers=_ver_tuple,
-        mask=0x3F,
-        flags=0x0,
-        OS=0x4,
-        fileType=0x1,
-        subtype=0x0,
-        date=(0, 0),
-    ),
-    kids=[
-        StringFileInfo(
-            [
-                StringTable(
-                    "040904B0",
-                    [
-                        StringStruct("Comments", "Youtube-dlc_x86 Command Line Interface."),
-                        StringStruct("CompanyName", "theidel@uni-bremen.de"),
-                        StringStruct("FileDescription", FILE_DESCRIPTION),
-                        StringStruct("FileVersion", version),
-                        StringStruct("InternalName", "youtube-dlc_x86"),
-                        StringStruct(
-                            "LegalCopyright",
-                            "theidel@uni-bremen.de | UNLICENSE",
-                        ),
-                        StringStruct("OriginalFilename", "youtube-dlc_x86.exe"),
-                        StringStruct("ProductName", "Youtube-dlc_x86"),
-                        StringStruct("ProductVersion", version + "_x86 | git.io/JUGsM"),
-                    ],
-                )
-            ]
-        ),
-        VarFileInfo([VarStruct("Translation", [0, 1200])])
-    ]
-)
-
-PyInstaller.__main__.run([
-    '--name=youtube-dlc_x86',
-    '--onefile',
-    '--icon=win/icon/cloud.ico',
-    'youtube_dlc/__main__.py',
-])
-SetVersion('dist/youtube-dlc_x86.exe', version_file)

requirements.txt

@@ -0,0 +1 @@
+mutagen

scripts/update-version-workflow.py

@@ -1,44 +0,0 @@
-from __future__ import unicode_literals
-from datetime import datetime
-# import urllib.request
-
-# response = urllib.request.urlopen('https://blackjack4494.github.io/youtube-dlc/update/LATEST_VERSION')
-# _LATEST_VERSION = response.read().decode('utf-8')
-
-exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
-
-_LATEST_VERSION = locals()['__version__']
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))
-
-print('::set-output name=ytdlc_version::' + version)
-
-file_version_py = open('youtube_dlc/version.py', 'rt')
-data = file_version_py.read()
-data = data.replace(locals()['__version__'], version)
-file_version_py.close()
-file_version_py = open('youtube_dlc/version.py', 'wt')
-file_version_py.write(data)
-file_version_py.close()

scripts/update-version.py

@@ -1,33 +0,0 @@
-# Unused
-
-from __future__ import unicode_literals
-from datetime import datetime
-import urllib.request
-
-response = urllib.request.urlopen('https://blackjack4494.github.io/youtube-dlc/update/LATEST_VERSION')
-
-_LATEST_VERSION = response.read().decode('utf-8')
-
-_OLD_VERSION = _LATEST_VERSION.rsplit("-", 1)
-
-if len(_OLD_VERSION) > 0:
-    old_ver = _OLD_VERSION[0]
-
-old_rev = ''
-if len(_OLD_VERSION) > 1:
-    old_rev = _OLD_VERSION[1]
-
-now = datetime.now()
-# ver = f'{datetime.today():%Y.%m.%d}'
-ver = now.strftime("%Y.%m.%d")
-rev = ''
-
-if old_ver == ver:
-    if old_rev:
-        rev = int(old_rev) + 1
-    else:
-        rev = 1
-
-_SEPARATOR = '-'
-
-version = _SEPARATOR.join(filter(None, [ver, str(rev)]))

setup.cfg

@@ -2,5 +2,5 @@
 universal = True
 
 [flake8]
-exclude = youtube_dlc/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv,devscripts/create-github-release.py,devscripts/release.sh,devscripts/show-downloads-statistics.py,scripts/update-version.py
+exclude = youtube_dlc/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv,devscripts/create-github-release.py,devscripts/release.sh,devscripts/show-downloads-statistics.py
 ignore = E402,E501,E731,E741,W503

setup.py

@@ -7,12 +7,21 @@ import warnings
 import sys
 from distutils.spawn import spawn
 
+
 # Get the version from youtube_dlc/version.py without importing the package
 exec(compile(open('youtube_dlc/version.py').read(),
              'youtube_dlc/version.py', 'exec'))
 
-DESCRIPTION = 'Media downloader supporting various sites such as youtube'
-LONG_DESCRIPTION = 'Command-line program to download videos from YouTube.com and other video sites. Based on a more active community fork.'
+
+DESCRIPTION = 'Command-line program to download videos from YouTube.com and many other other video platforms.'
+
+LONG_DESCRIPTION = '\n\n'.join((
+    'Official repository: <https://github.com/pukkandan/yt-dlp>',
+    '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
+    open("README.md", "r", encoding="utf-8").read()))
+
+REQUIREMENTS = ['mutagen']
+
 
 if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
     print("inv")
@@ -37,10 +46,8 @@ else:
     params = {
         'data_files': data_files,
     }
-    #if setuptools_available:
     params['entry_points'] = {'console_scripts': ['youtube-dlc = youtube_dlc:main']}
-    #else:
-    #    params['scripts'] = ['bin/youtube-dlc']
+
 
 class build_lazy_extractors(Command):
     description = 'Build the extractor lazy loading module'
@@ -58,20 +65,26 @@ class build_lazy_extractors(Command):
             dry_run=self.dry_run,
         )
 
+
+packages = find_packages(exclude=("youtube_dl", "test", "ytdlp_plugins"))
+
 setup(
-    name="youtube_dlc",
+    name="yt-dlp",
     version=__version__,
-    maintainer="Tom-Oliver Heidel",
-    maintainer_email="theidel@uni-bremen.de",
+    maintainer="pukkandan",
+    maintainer_email="pukkandan@gmail.com",
     description=DESCRIPTION,
     long_description=LONG_DESCRIPTION,
-    # long_description_content_type="text/markdown",
-    url="https://github.com/pukkandan/yt-dlc",
-    packages=find_packages(exclude=("youtube_dl","test",)),
-	#packages=[
-    #    'youtube_dlc',
-    #    'youtube_dlc.extractor', 'youtube_dlc.downloader',
-    #    'youtube_dlc.postprocessor'],
+    long_description_content_type="text/markdown",
+    url="https://github.com/pukkandan/yt-dlp",
+    packages=packages,
+    install_requires=REQUIREMENTS,
+    project_urls={
+        'Documentation': 'https://github.com/pukkandan/yt-dlp#yt-dlp',
+        'Source': 'https://github.com/pukkandan/yt-dlp',
+        'Tracker': 'https://github.com/pukkandan/yt-dlp/issues',
+        #'Funding': 'https://donate.pypi.org',
+    },
     classifiers=[
 	    "Topic :: Multimedia :: Video",
         "Development Status :: 5 - Production/Stable",

test/test_YoutubeDL.py

@@ -637,13 +637,20 @@ class TestYoutubeDL(unittest.TestCase):
             'title2': '%PATH%',
         }
 
-        def fname(templ):
-            ydl = YoutubeDL({'outtmpl': templ})
+        def fname(templ, na_placeholder='NA'):
+            params = {'outtmpl': templ}
+            if na_placeholder != 'NA':
+                params['outtmpl_na_placeholder'] = na_placeholder
+            ydl = YoutubeDL(params)
             return ydl.prepare_filename(info)
         self.assertEqual(fname('%(id)s.%(ext)s'), '1234.mp4')
         self.assertEqual(fname('%(id)s-%(width)s.%(ext)s'), '1234-NA.mp4')
-        # Replace missing fields with 'NA'
-        self.assertEqual(fname('%(uploader_date)s-%(id)s.%(ext)s'), 'NA-1234.mp4')
+        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(id)s.%(ext)s'
+        # Replace missing fields with 'NA' by default
+        self.assertEqual(fname(NA_TEST_OUTTMPL), 'NA-NA-1234.mp4')
+        # Or by provided placeholder
+        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder='none'), 'none-none-1234.mp4')
+        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder=''), '--1234.mp4')
         self.assertEqual(fname('%(height)d.%(ext)s'), '1080.mp4')
         self.assertEqual(fname('%(height)6d.%(ext)s'), '  1080.mp4')
         self.assertEqual(fname('%(height)-6d.%(ext)s'), '1080  .mp4')

test/test_post_hooks.py

@@ -8,11 +8,11 @@ import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from test.helper import get_params, try_rm
-import youtube_dl.YoutubeDL
-from youtube_dl.utils import DownloadError
+import youtube_dlc.YoutubeDL
+from youtube_dlc.utils import DownloadError
 
 
-class YoutubeDL(youtube_dl.YoutubeDL):
+class YoutubeDL(youtube_dlc.YoutubeDL):
     def __init__(self, *args, **kwargs):
         super(YoutubeDL, self).__init__(*args, **kwargs)
         self.to_stderr = self.to_screen

test/test_postprocessors.py

@@ -8,10 +8,16 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from youtube_dlc.postprocessor import MetadataFromTitlePP
+from youtube_dlc.postprocessor import MetadataFromFieldPP, MetadataFromTitlePP
+
+
+class TestMetadataFromField(unittest.TestCase):
+    def test_format_to_regex(self):
+        pp = MetadataFromFieldPP(None, ['title:%(title)s - %(artist)s'])
+        self.assertEqual(pp._data[0]['regex'], r'(?P<title>[^\r\n]+)\ \-\ (?P<artist>[^\r\n]+)')
 
 
 class TestMetadataFromTitle(unittest.TestCase):
     def test_format_to_regex(self):
         pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
-        self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+        self.assertEqual(pp._titleregex, r'(?P<title>[^\r\n]+)\ \-\ (?P<artist>[^\r\n]+)')

test/test_unicode_literals.py

@@ -15,8 +15,6 @@ IGNORED_FILES = [
     'setup.py',  # http://bugs.python.org/issue13943
     'conf.py',
     'buildserver.py',
-    'pyinst.py',
-    'pyinst32.py',
 ]
 
 IGNORED_DIRS = [

win/ver.txt

@@ -1,45 +0,0 @@
-# UTF-8
-#
-# For more details about fixed file info 'ffi' see:
-# http://msdn.microsoft.com/en-us/library/ms646997.aspx
-VSVersionInfo(
-  ffi=FixedFileInfo(
-    # filevers and prodvers should be always a tuple with four items: (1, 2, 3, 4)
-    # Set not needed items to zero 0.
-    filevers=(16, 9, 2020, 0),
-    prodvers=(16, 9, 2020, 0),
-    # Contains a bitmask that specifies the valid bits 'flags'r
-    mask=0x3f,
-    # Contains a bitmask that specifies the Boolean attributes of the file.
-    flags=0x0,
-    # The operating system for which this file was designed.
-    # 0x4 - NT and there is no need to change it.
-    # OS=0x40004,
-	OS=0x4,
-    # The general type of file.
-    # 0x1 - the file is an application.
-    fileType=0x1,
-    # The function of the file.
-    # 0x0 - the function is not defined for this fileType
-    subtype=0x0,
-    # Creation date and time stamp.
-    date=(0, 0)
-    ),
-  kids=[
-    StringFileInfo(
-      [
-      StringTable(
-        u'040904B0',
-        [StringStruct(u'Comments', u'Youtube-dlc Command Line Interface.'),
-	StringStruct(u'CompanyName', u'theidel@uni-bremen.de'),
-        StringStruct(u'FileDescription', u'Media Downloader'),
-        StringStruct(u'FileVersion', u'16.9.2020.0'),
-        StringStruct(u'InternalName', u'youtube-dlc'),
-        StringStruct(u'LegalCopyright', u'theidel@uni-bremen.de | UNLICENSE'),
-        StringStruct(u'OriginalFilename', u'youtube-dlc.exe'),
-        StringStruct(u'ProductName', u'Youtube-dlc'),
-        StringStruct(u'ProductVersion', u'16.9.2020.0 | git.io/JUGsM')])
-      ]), 
-    VarFileInfo([VarStruct(u'Translation', [0, 1200])])
-  ]
-)

youtube-dlc.cmd

@@ -1 +1 @@
-py "%~dp0youtube_dl\__main__.py"
+@py "%~dp0youtube_dlc\__main__.py" %*

youtube_dlc/__init__.py

@@ -8,17 +8,17 @@ __license__ = 'Public Domain'
 import codecs
 import io
 import os
-import re
 import random
+import re
 import sys
 
 
 from .options import (
     parseOpts,
+    _remux_formats,
 )
 from .compat import (
     compat_getpass,
-    compat_shlex_split,
     workaround_optparse_bug9161,
 )
 from .utils import (
@@ -26,11 +26,13 @@ from .utils import (
     decodeOption,
     DEFAULT_OUTTMPL,
     DownloadError,
+    ExistingVideoReached,
     expand_path,
     match_filter_func,
     MaxDownloadsReached,
     preferredencoding,
     read_batch_urls,
+    RejectedVideoReached,
     SameFileError,
     setproctitle,
     std_headers,
@@ -44,6 +46,7 @@ from .downloader import (
 from .extractor import gen_extractors, list_extractors
 from .extractor.common import InfoExtractor
 from .extractor.adobepass import MSO_INFO
+from .postprocessor.metadatafromfield import MetadataFromFieldPP
 from .YoutubeDL import YoutubeDL
 
 
@@ -68,14 +71,7 @@ def _real_main(argv=None):
         std_headers['Referer'] = opts.referer
 
     # Custom HTTP headers
-    if opts.headers is not None:
-        for h in opts.headers:
-            if ':' not in h:
-                parser.error('wrong header formatting, it should be key:value, not "%s"' % h)
-            key, value = h.split(':', 1)
-            if opts.verbose:
-                write_string('[debug] Adding header from command line option %s:%s\n' % (key, value))
-            std_headers[key] = value
+    std_headers.update(opts.headers)
 
     # Dump user agent
     if opts.dump_user_agent:
@@ -214,12 +210,15 @@ def _real_main(argv=None):
         opts.audioquality = opts.audioquality.strip('k').strip('K')
         if not opts.audioquality.isdigit():
             parser.error('invalid audio quality specified')
-    if opts.remuxvideo is not None:
-        if opts.remuxvideo not in ['mp4', 'mkv']:
-            parser.error('invalid video container format specified')
     if opts.recodevideo is not None:
-        if opts.recodevideo not in ['mp4', 'flv', 'webm', 'ogg', 'mkv', 'avi']:
+        if opts.recodevideo not in _remux_formats:
             parser.error('invalid video recode format specified')
+    if opts.remuxvideo and opts.recodevideo:
+        opts.remuxvideo = None
+        write_string('WARNING: --remux-video is ignored since --recode-video was given\n', out=sys.stderr)
+    if opts.remuxvideo is not None:
+        if opts.remuxvideo not in _remux_formats:
+            parser.error('invalid video remux format specified')
     if opts.convertsubtitles is not None:
         if opts.convertsubtitles not in ['srt', 'vtt', 'ass', 'lrc']:
             parser.error('invalid subtitle format specified')
@@ -250,20 +249,30 @@ def _real_main(argv=None):
         parser.error('Cannot download a video and extract audio into the same'
                      ' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
                      ' template'.format(outtmpl))
+
     for f in opts.format_sort:
         if re.match(InfoExtractor.FormatSort.regex, f) is None:
             parser.error('invalid format sort string "%s" specified' % f)
 
+    if opts.metafromfield is None:
+        opts.metafromfield = []
+    if opts.metafromtitle is not None:
+        opts.metafromfield.append('title:%s' % opts.metafromtitle)
+    for f in opts.metafromfield:
+        if re.match(MetadataFromFieldPP.regex, f) is None:
+            parser.error('invalid format string "%s" specified for --parse-metadata' % f)
+
     any_getting = opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
     any_printing = opts.print_json
     download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive
 
     # PostProcessors
     postprocessors = []
-    if opts.metafromtitle:
+    if opts.metafromfield:
         postprocessors.append({
-            'key': 'MetadataFromTitle',
-            'titleformat': opts.metafromtitle
+            'key': 'MetadataFromField',
+            'formats': opts.metafromfield,
+            'when': 'beforedl'
         })
     if opts.extractaudio:
         postprocessors.append({
@@ -324,32 +333,24 @@ def _real_main(argv=None):
             'force': opts.sponskrub_force,
             'ignoreerror': opts.sponskrub is None,
         })
-    # Please keep ExecAfterDownload towards the bottom as it allows the user to modify the final file in any way.
-    # So if the user is able to remove the file before your postprocessor runs it might cause a few problems.
+    # ExecAfterDownload must be the last PP
     if opts.exec_cmd:
         postprocessors.append({
             'key': 'ExecAfterDownload',
             'exec_cmd': opts.exec_cmd,
+            'when': 'aftermove'
         })
-    external_downloader_args = None
-    if opts.external_downloader_args:
-        external_downloader_args = compat_shlex_split(opts.external_downloader_args)
 
-    postprocessor_args = {}
-    if opts.postprocessor_args is not None:
-        for string in opts.postprocessor_args:
-            mobj = re.match(r'(?P<pp>\w+):(?P<args>.*)$', string)
-            if mobj is None:
-                if 'sponskrub' not in postprocessor_args:  # for backward compatibility
-                    postprocessor_args['sponskrub'] = []
-                    if opts.verbose:
-                        write_string('[debug] Adding postprocessor args from command line option sponskrub:\n')
-                pp_name, pp_args = 'default', string
-            else:
-                pp_name, pp_args = mobj.group('pp').lower(), mobj.group('args')
-            if opts.verbose:
-                write_string('[debug] Adding postprocessor args from command line option %s:%s\n' % (pp_name, pp_args))
-            postprocessor_args[pp_name] = compat_shlex_split(pp_args)
+    _args_compat_warning = 'WARNING: %s given without specifying name. The arguments will be given to all %s\n'
+    if 'default' in opts.external_downloader_args:
+        write_string(_args_compat_warning % ('--external-downloader-args', 'external downloaders'), out=sys.stderr),
+
+    if 'default-compat' in opts.postprocessor_args and 'default' not in opts.postprocessor_args:
+        write_string(_args_compat_warning % ('--post-processor-args', 'post-processors'), out=sys.stderr),
+        opts.postprocessor_args.setdefault('sponskrub', [])
+        opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
+
+    audio_ext = opts.audioformat if (opts.extractaudio and opts.audioformat != 'best') else None
 
     match_filter = (
         None if opts.match_filter is None
@@ -388,6 +389,8 @@ def _real_main(argv=None):
         'listformats': opts.listformats,
         'listformats_table': opts.listformats_table,
         'outtmpl': outtmpl,
+        'outtmpl_na_placeholder': opts.outtmpl_na_placeholder,
+        'paths': opts.paths,
         'autonumber_size': opts.autonumber_size,
         'autonumber_start': opts.autonumber_start,
         'restrictfilenames': opts.restrictfilenames,
@@ -416,7 +419,8 @@ def _real_main(argv=None):
         'updatetime': opts.updatetime,
         'writedescription': opts.writedescription,
         'writeannotations': opts.writeannotations,
-        'writeinfojson': opts.writeinfojson,
+        'writeinfojson': opts.writeinfojson or opts.getcomments,
+        'getcomments': opts.getcomments,
         'writethumbnail': opts.writethumbnail,
         'write_all_thumbnails': opts.write_all_thumbnails,
         'writelink': opts.writelink,
@@ -449,6 +453,7 @@ def _real_main(argv=None):
         'age_limit': opts.age_limit,
         'download_archive': download_archive_fn,
         'break_on_existing': opts.break_on_existing,
+        'break_on_reject': opts.break_on_reject,
         'cookiefile': opts.cookiefile,
         'nocheckcertificate': opts.no_check_certificate,
         'prefer_insecure': opts.prefer_insecure,
@@ -466,6 +471,7 @@ def _real_main(argv=None):
         'extract_flat': opts.extract_flat,
         'mark_watched': opts.mark_watched,
         'merge_output_format': opts.merge_output_format,
+        'final_ext': opts.recodevideo or opts.remuxvideo or audio_ext,
         'postprocessors': postprocessors,
         'fixup': opts.fixup,
         'source_address': opts.source_address,
@@ -482,8 +488,8 @@ def _real_main(argv=None):
         'ffmpeg_location': opts.ffmpeg_location,
         'hls_prefer_native': opts.hls_prefer_native,
         'hls_use_mpegts': opts.hls_use_mpegts,
-        'external_downloader_args': external_downloader_args,
-        'postprocessor_args': postprocessor_args,
+        'external_downloader_args': opts.external_downloader_args,
+        'postprocessor_args': opts.postprocessor_args,
         'cn_verification_proxy': opts.cn_verification_proxy,
         'geo_verification_proxy': opts.geo_verification_proxy,
         'config_location': opts.config_location,
@@ -519,8 +525,8 @@ def _real_main(argv=None):
                 retcode = ydl.download_with_info_file(expand_path(opts.load_info_filename))
             else:
                 retcode = ydl.download(all_urls)
-        except MaxDownloadsReached:
-            ydl.to_screen('--max-download limit reached, aborting.')
+        except (MaxDownloadsReached, ExistingVideoReached, RejectedVideoReached):
+            ydl.to_screen('Aborting remaining downloads')
             retcode = 101
 
     sys.exit(retcode)

youtube_dlc/downloader/external.py

@@ -95,7 +95,8 @@ class ExternalFD(FileDownloader):
         return cli_valueless_option(self.params, command_option, param, expected_value)
 
     def _configuration_args(self, default=[]):
-        return cli_configuration_args(self.params, 'external_downloader_args', default)
+        return cli_configuration_args(
+            self.params, 'external_downloader_args', self.get_basename(), default)[0]
 
     def _call_downloader(self, tmpfilename, info_dict):
         """ Either overwrite this or implement _make_cmd """
@@ -232,7 +233,7 @@ class FFmpegFD(ExternalFD):
         url = info_dict['url']
         ffpp = FFmpegPostProcessor(downloader=self)
         if not ffpp.available:
-            self.report_error('m3u8 download detected but ffmpeg or avconv could not be found. Please install one.')
+            self.report_error('m3u8 download detected but ffmpeg could not be found. Please install one.')
             return False
         ffpp.check_version()
 

youtube_dlc/extractor/__init__.py

@@ -1,13 +1,20 @@
 from __future__ import unicode_literals
 
+from ..utils import load_plugins
+
 try:
     from .lazy_extractors import *
     from .lazy_extractors import _ALL_CLASSES
     _LAZY_LOADER = True
+    _PLUGIN_CLASSES = []
 except ImportError:
     _LAZY_LOADER = False
+
+if not _LAZY_LOADER:
     from .extractors import *
 
+    _PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
+
     _ALL_CLASSES = [
         klass
         for name, klass in globals().items()

youtube_dlc/extractor/adn.py

@@ -10,6 +10,7 @@ import random
 from .common import InfoExtractor
 from ..aes import aes_cbc_decrypt
 from ..compat import (
+    compat_HTTPError,
     compat_b64decode,
     compat_ord,
 )
@@ -18,11 +19,13 @@ from ..utils import (
     bytes_to_long,
     ExtractorError,
     float_or_none,
+    int_or_none,
     intlist_to_bytes,
     long_to_bytes,
     pkcs1pad,
     strip_or_none,
-    urljoin,
+    try_get,
+    unified_strdate,
 )
 
 
@@ -31,16 +34,27 @@ class ADNIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?animedigitalnetwork\.fr/video/[^/]+/(?P<id>\d+)'
     _TEST = {
         'url': 'http://animedigitalnetwork.fr/video/blue-exorcist-kyoto-saga/7778-episode-1-debut-des-hostilites',
-        'md5': 'e497370d847fd79d9d4c74be55575c7a',
+        'md5': '0319c99885ff5547565cacb4f3f9348d',
         'info_dict': {
             'id': '7778',
             'ext': 'mp4',
-            'title': 'Blue Exorcist - Kyôto Saga - Épisode 1',
+            'title': 'Blue Exorcist - Kyôto Saga - Episode 1',
             'description': 'md5:2f7b5aa76edbc1a7a92cedcda8a528d5',
+            'series': 'Blue Exorcist - Kyôto Saga',
+            'duration': 1467,
+            'release_date': '20170106',
+            'comment_count': int,
+            'average_rating': float,
+            'season_number': 2,
+            'episode': 'Début des hostilités',
+            'episode_number': 1,
         }
     }
+
     _BASE_URL = 'http://animedigitalnetwork.fr'
-    _RSA_KEY = (0xc35ae1e4356b65a73b551493da94b8cb443491c0aa092a357a5aee57ffc14dda85326f42d716e539a34542a0d3f363adf16c5ec222d713d5997194030ee2e4f0d1fb328c01a81cf6868c090d50de8e169c6b13d1675b9eeed1cbc51e1fffca9b38af07f37abd790924cd3bee59d0257cfda4fe5f3f0534877e21ce5821447d1b, 65537)
+    _API_BASE_URL = 'https://gw.api.animedigitalnetwork.fr/'
+    _PLAYER_BASE_URL = _API_BASE_URL + 'player/'
+    _RSA_KEY = (0x9B42B08905199A5CCE2026274399CA560ECB209EE9878A708B1C0812E1BB8CB5D1FB7441861147C1A1F2F3A0476DD63A9CAC20D3E983613346850AA6CB38F16DC7D720FD7D86FC6E5B3D5BBC72E14CD0BF9E869F2CEA2CCAD648F1DCE38F1FF916CEFB2D339B64AA0264372344BC775E265E8A852F88144AB0BD9AA06C1A4ABB, 65537)
     _POS_ALIGN_MAP = {
         'start': 1,
         'end': 3,
@@ -54,26 +68,24 @@ class ADNIE(InfoExtractor):
     def _ass_subtitles_timecode(seconds):
         return '%01d:%02d:%02d.%02d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 100)
 
-    def _get_subtitles(self, sub_path, video_id):
-        if not sub_path:
+    def _get_subtitles(self, sub_url, video_id):
+        if not sub_url:
             return None
 
         enc_subtitles = self._download_webpage(
-            urljoin(self._BASE_URL, sub_path),
-            video_id, 'Downloading subtitles location', fatal=False) or '{}'
+            sub_url, video_id, 'Downloading subtitles location', fatal=False) or '{}'
         subtitle_location = (self._parse_json(enc_subtitles, video_id, fatal=False) or {}).get('location')
         if subtitle_location:
             enc_subtitles = self._download_webpage(
-                urljoin(self._BASE_URL, subtitle_location),
-                video_id, 'Downloading subtitles data', fatal=False,
-                headers={'Origin': 'https://animedigitalnetwork.fr'})
+                subtitle_location, video_id, 'Downloading subtitles data',
+                fatal=False, headers={'Origin': 'https://animedigitalnetwork.fr'})
         if not enc_subtitles:
             return None
 
         # http://animedigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
         dec_subtitles = intlist_to_bytes(aes_cbc_decrypt(
             bytes_to_intlist(compat_b64decode(enc_subtitles[24:])),
-            bytes_to_intlist(binascii.unhexlify(self._K + '4b8ef13ec1872730')),
+            bytes_to_intlist(binascii.unhexlify(self._K + 'ab9f52f5baae7c72')),
             bytes_to_intlist(compat_b64decode(enc_subtitles[:24]))
         ))
         subtitles_json = self._parse_json(
@@ -119,59 +131,76 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        player_config = self._parse_json(self._search_regex(
-            r'playerConfig\s*=\s*({.+});', webpage,
-            'player config', default='{}'), video_id, fatal=False)
-        if not player_config:
-            config_url = urljoin(self._BASE_URL, self._search_regex(
-                r'(?:id="player"|class="[^"]*adn-player-container[^"]*")[^>]+data-url="([^"]+)"',
-                webpage, 'config url'))
-            player_config = self._download_json(
-                config_url, video_id,
-                'Downloading player config JSON metadata')['player']
+        video_base_url = self._PLAYER_BASE_URL + 'video/%s/' % video_id
+        player = self._download_json(
+            video_base_url + 'configuration', video_id,
+            'Downloading player config JSON metadata')['player']
+        options = player['options']
 
-        video_info = {}
-        video_info_str = self._search_regex(
-            r'videoInfo\s*=\s*({.+});', webpage,
-            'video info', fatal=False)
-        if video_info_str:
-            video_info = self._parse_json(
-                video_info_str, video_id, fatal=False) or {}
+        user = options['user']
+        if not user.get('hasAccess'):
+            raise ExtractorError(
+                'This video is only available for paying users', expected=True)
+            # self.raise_login_required() # FIXME: Login is not implemented
 
-        options = player_config.get('options') or {}
-        metas = options.get('metas') or {}
-        links = player_config.get('links') or {}
-        sub_path = player_config.get('subtitles')
-        error = None
-        if not links:
-            links_url = player_config.get('linksurl') or options['videoUrl']
-            token = options['token']
-            self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
-            message = bytes_to_intlist(json.dumps({
-                'k': self._K,
-                'e': 60,
-                't': token,
-            }))
+        token = self._download_json(
+            user.get('refreshTokenUrl') or (self._PLAYER_BASE_URL + 'refresh/token'),
+            video_id, 'Downloading access token', headers={
+                'x-player-refresh-token': user['refreshToken']
+            }, data=b'')['token']
+
+        links_url = try_get(options, lambda x: x['video']['url']) or (video_base_url + 'link')
+        self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
+        message = bytes_to_intlist(json.dumps({
+            'k': self._K,
+            't': token,
+        }))
+
+        # Sometimes authentication fails for no good reason, retry with
+        # a different random padding
+        links_data = None
+        for _ in range(3):
             padded_message = intlist_to_bytes(pkcs1pad(message, 128))
             n, e = self._RSA_KEY
             encrypted_message = long_to_bytes(pow(bytes_to_long(padded_message), e, n))
             authorization = base64.b64encode(encrypted_message).decode()
-            links_data = self._download_json(
-                urljoin(self._BASE_URL, links_url), video_id,
-                'Downloading links JSON metadata', headers={
-                    'Authorization': 'Bearer ' + authorization,
-                })
-            links = links_data.get('links') or {}
-            metas = metas or links_data.get('meta') or {}
-            sub_path = sub_path or links_data.get('subtitles') or \
-                'index.php?option=com_vodapi&task=subtitles.getJSON&format=json&id=' + video_id
-            sub_path += '&token=' + token
-            error = links_data.get('error')
-        title = metas.get('title') or video_info['title']
+
+            try:
+                links_data = self._download_json(
+                    links_url, video_id, 'Downloading links JSON metadata', headers={
+                        'X-Player-Token': authorization
+                    }, query={
+                        'freeWithAds': 'true',
+                        'adaptive': 'false',
+                        'withMetadata': 'true',
+                        'source': 'Web'
+                    })
+                break
+            except ExtractorError as e:
+                if not isinstance(e.cause, compat_HTTPError):
+                    raise e
+
+                if e.cause.code == 401:
+                    # This usually goes away with a different random pkcs1pad, so retry
+                    continue
+
+                error = self._parse_json(e.cause.read(), video_id)
+                message = error.get('message')
+                if e.cause.code == 403 and error.get('code') == 'player-bad-geolocation-country':
+                    self.raise_geo_restricted(msg=message)
+                else:
+                    raise ExtractorError(message)
+        else:
+            raise ExtractorError('Giving up retrying')
+
+        links = links_data.get('links') or {}
+        metas = links_data.get('metadata') or {}
+        sub_url = (links.get('subtitles') or {}).get('all')
+        video_info = links_data.get('video') or {}
+        title = metas['title']
 
         formats = []
-        for format_id, qualities in links.items():
+        for format_id, qualities in (links.get('streaming') or {}).items():
             if not isinstance(qualities, dict):
                 continue
             for quality, load_balancer_url in qualities.items():
@@ -189,19 +218,26 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                     for f in m3u8_formats:
                         f['language'] = 'fr'
                 formats.extend(m3u8_formats)
-        if not error:
-            error = options.get('error')
-        if not formats and error:
-            raise ExtractorError('%s said: %s' % (self.IE_NAME, error), expected=True)
         self._sort_formats(formats)
 
+        video = (self._download_json(
+            self._API_BASE_URL + 'video/%s' % video_id, video_id,
+            'Downloading additional video metadata', fatal=False) or {}).get('video') or {}
+        show = video.get('show') or {}
+
         return {
             'id': video_id,
             'title': title,
-            'description': strip_or_none(metas.get('summary') or video_info.get('resume')),
-            'thumbnail': video_info.get('image'),
+            'description': strip_or_none(metas.get('summary') or video.get('summary')),
+            'thumbnail': video_info.get('image') or player.get('image'),
             'formats': formats,
-            'subtitles': self.extract_subtitles(sub_path, video_id),
-            'episode': metas.get('subtitle') or video_info.get('videoTitle'),
-            'series': video_info.get('playlistTitle'),
+            'subtitles': self.extract_subtitles(sub_url, video_id),
+            'episode': metas.get('subtitle') or video.get('name'),
+            'episode_number': int_or_none(video.get('shortNumber')),
+            'series': show.get('title'),
+            'season_number': int_or_none(video.get('season')),
+            'duration': int_or_none(video_info.get('duration') or video.get('duration')),
+            'release_date': unified_strdate(video.get('releaseDate')),
+            'average_rating': float_or_none(video.get('rating') or metas.get('rating')),
+            'comment_count': int_or_none(video.get('commentsCount')),
         }

youtube_dlc/extractor/aenetworks.py

@@ -256,7 +256,7 @@ class AENetworksShowIE(AENetworksListBaseIE):
             'title': 'Ancient Aliens',
             'description': 'md5:3f6d74daf2672ff3ae29ed732e37ea7f',
         },
-        'playlist_mincount': 168,
+        'playlist_mincount': 150,
     }]
     _RESOURCE = 'series'
     _ITEMS_KEY = 'episodes'

youtube_dlc/extractor/aljazeera.py

@@ -1,13 +1,16 @@
 from __future__ import unicode_literals
 
+import json
+import re
+
 from .common import InfoExtractor
 
 
 class AlJazeeraIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?:programmes|video)/.*?/(?P<id>[^/]+)\.html'
+    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?P<type>program/[^/]+|(?:feature|video)s)/\d{4}/\d{1,2}/\d{1,2}/(?P<id>[^/?&#]+)'
 
     _TESTS = [{
-        'url': 'http://www.aljazeera.com/programmes/the-slum/2014/08/deliverance-201482883754237240.html',
+        'url': 'https://www.aljazeera.com/program/episode/2014/9/19/deliverance',
         'info_dict': {
             'id': '3792260579001',
             'ext': 'mp4',
@@ -20,14 +23,34 @@ class AlJazeeraIE(InfoExtractor):
         'add_ie': ['BrightcoveNew'],
         'skip': 'Not accessible from Travis CI server',
     }, {
-        'url': 'http://www.aljazeera.com/video/news/2017/05/sierra-leone-709-carat-diamond-auctioned-170511100111930.html',
+        'url': 'https://www.aljazeera.com/videos/2017/5/11/sierra-leone-709-carat-diamond-to-be-auctioned-off',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.aljazeera.com/features/2017/8/21/transforming-pakistans-buses-into-art',
         'only_matching': True,
     }]
-    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/665003303001/default_default/index.html?videoId=%s'
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_default/index.html?videoId=%s'
 
     def _real_extract(self, url):
-        program_name = self._match_id(url)
-        webpage = self._download_webpage(url, program_name)
-        brightcove_id = self._search_regex(
-            r'RenderPagesVideo\(\'(.+?)\'', webpage, 'brightcove id')
-        return self.url_result(self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id, 'BrightcoveNew', brightcove_id)
+        post_type, name = re.match(self._VALID_URL, url).groups()
+        post_type = {
+            'features': 'post',
+            'program': 'episode',
+            'videos': 'video',
+        }[post_type.split('/')[0]]
+        video = self._download_json(
+            'https://www.aljazeera.com/graphql', name, query={
+                'operationName': 'SingleArticleQuery',
+                'variables': json.dumps({
+                    'name': name,
+                    'postType': post_type,
+                }),
+            }, headers={
+                'wp-site': 'aje',
+            })['data']['article']['video']
+        video_id = video['id']
+        account_id = video.get('accountId') or '665003303001'
+        player_id = video.get('playerId') or 'BkeSH5BDb'
+        return self.url_result(
+            self.BRIGHTCOVE_URL_TEMPLATE % (account_id, player_id, video_id),
+            'BrightcoveNew', video_id)

youtube_dlc/extractor/americastestkitchen.py

@@ -1,13 +1,16 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 
 from .common import InfoExtractor
 from ..utils import (
     clean_html,
+    int_or_none,
     try_get,
     unified_strdate,
+    unified_timestamp,
 )
 
 
@@ -22,8 +25,8 @@ class AmericasTestKitchenIE(InfoExtractor):
             'ext': 'mp4',
             'description': 'md5:64e606bfee910627efc4b5f050de92b3',
             'thumbnail': r're:^https?://',
-            'timestamp': 1523664000,
-            'upload_date': '20180414',
+            'timestamp': 1523318400,
+            'upload_date': '20180410',
             'release_date': '20180410',
             'series': "America's Test Kitchen",
             'season_number': 18,
@@ -33,6 +36,27 @@ class AmericasTestKitchenIE(InfoExtractor):
         'params': {
             'skip_download': True,
         },
+    }, {
+        # Metadata parsing behaves differently for newer episodes (705) as opposed to older episodes (582 above)
+        'url': 'https://www.americastestkitchen.com/episode/705-simple-chicken-dinner',
+        'md5': '06451608c57651e985a498e69cec17e5',
+        'info_dict': {
+            'id': '5fbe8c61bda2010001c6763b',
+            'title': 'Simple Chicken Dinner',
+            'ext': 'mp4',
+            'description': 'md5:eb68737cc2fd4c26ca7db30139d109e7',
+            'thumbnail': r're:^https?://',
+            'timestamp': 1610755200,
+            'upload_date': '20210116',
+            'release_date': '20210116',
+            'series': "America's Test Kitchen",
+            'season_number': 21,
+            'episode': 'Simple Chicken Dinner',
+            'episode_number': 3,
+        },
+        'params': {
+            'skip_download': True,
+        },
     }, {
         'url': 'https://www.americastestkitchen.com/videos/3420-pan-seared-salmon',
         'only_matching': True,
@@ -60,7 +84,76 @@ class AmericasTestKitchenIE(InfoExtractor):
             'url': 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % video['zypeId'],
             'ie_key': 'Zype',
             'description': clean_html(video.get('description')),
+            'timestamp': unified_timestamp(video.get('publishDate')),
             'release_date': unified_strdate(video.get('publishDate')),
+            'episode_number': int_or_none(episode.get('number')),
+            'season_number': int_or_none(episode.get('season')),
             'series': try_get(episode, lambda x: x['show']['title']),
             'episode': episode.get('title'),
         }
+
+
+class AmericasTestKitchenSeasonIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?(?P<show>americastestkitchen|cookscountry)\.com/episodes/browse/season_(?P<id>\d+)'
+    _TESTS = [{
+        # ATK Season
+        'url': 'https://www.americastestkitchen.com/episodes/browse/season_1',
+        'info_dict': {
+            'id': 'season_1',
+            'title': 'Season 1',
+        },
+        'playlist_count': 13,
+    }, {
+        # Cooks Country Season
+        'url': 'https://www.cookscountry.com/episodes/browse/season_12',
+        'info_dict': {
+            'id': 'season_12',
+            'title': 'Season 12',
+        },
+        'playlist_count': 13,
+    }]
+
+    def _real_extract(self, url):
+        show_name, season_number = re.match(self._VALID_URL, url).groups()
+        season_number = int(season_number)
+
+        slug = 'atk' if show_name == 'americastestkitchen' else 'cco'
+
+        season = 'Season %d' % season_number
+
+        season_search = self._download_json(
+            'https://y1fnzxui30-dsn.algolia.net/1/indexes/everest_search_%s_season_desc_production' % slug,
+            season, headers={
+                'Origin': 'https://www.%s.com' % show_name,
+                'X-Algolia-API-Key': '8d504d0099ed27c1b73708d22871d805',
+                'X-Algolia-Application-Id': 'Y1FNZXUI30',
+            }, query={
+                'facetFilters': json.dumps([
+                    'search_season_list:' + season,
+                    'search_document_klass:episode',
+                    'search_show_slug:' + slug,
+                ]),
+                'attributesToRetrieve': 'description,search_%s_episode_number,search_document_date,search_url,title' % slug,
+                'attributesToHighlight': '',
+                'hitsPerPage': 1000,
+            })
+
+        def entries():
+            for episode in (season_search.get('hits') or []):
+                search_url = episode.get('search_url')
+                if not search_url:
+                    continue
+                yield {
+                    '_type': 'url',
+                    'url': 'https://www.%s.com%s' % (show_name, search_url),
+                    'id': try_get(episode, lambda e: e['objectID'].split('_')[-1]),
+                    'title': episode.get('title'),
+                    'description': episode.get('description'),
+                    'timestamp': unified_timestamp(episode.get('search_document_date')),
+                    'season_number': season_number,
+                    'episode_number': int_or_none(episode.get('search_%s_episode_number' % slug)),
+                    'ie_key': AmericasTestKitchenIE.ie_key(),
+                }
+
+        return self.playlist_result(
+            entries(), 'season_%d' % season_number, season)

youtube_dlc/extractor/animeondemand.py

@@ -116,8 +116,6 @@ class AnimeOnDemandIE(InfoExtractor):
             r'(?s)<div[^>]+itemprop="description"[^>]*>(.+?)</div>',
             webpage, 'anime description', default=None)
 
-        entries = []
-
         def extract_info(html, video_id, num=None):
             title, description = [None] * 2
             formats = []
@@ -233,7 +231,7 @@ class AnimeOnDemandIE(InfoExtractor):
                 self._sort_formats(info['formats'])
                 f = common_info.copy()
                 f.update(info)
-                entries.append(f)
+                yield f
 
             # Extract teaser/trailer only when full episode is not available
             if not info['formats']:
@@ -247,7 +245,7 @@ class AnimeOnDemandIE(InfoExtractor):
                         'title': m.group('title'),
                         'url': urljoin(url, m.group('href')),
                     })
-                    entries.append(f)
+                    yield f
 
         def extract_episodes(html):
             for num, episode_html in enumerate(re.findall(
@@ -275,7 +273,8 @@ class AnimeOnDemandIE(InfoExtractor):
                     'episode_number': episode_number,
                 }
 
-                extract_entries(episode_html, video_id, common_info)
+                for e in extract_entries(episode_html, video_id, common_info):
+                    yield e
 
         def extract_film(html, video_id):
             common_info = {
@@ -283,11 +282,18 @@ class AnimeOnDemandIE(InfoExtractor):
                 'title': anime_title,
                 'description': anime_description,
             }
-            extract_entries(html, video_id, common_info)
+            for e in extract_entries(html, video_id, common_info):
+                yield e
 
-        extract_episodes(webpage)
+        def entries():
+            has_episodes = False
+            for e in extract_episodes(webpage):
+                has_episodes = True
+                yield e
 
-        if not entries:
-            extract_film(webpage, anime_id)
+            if not has_episodes:
+                for e in extract_film(webpage, anime_id):
+                    yield e
 
-        return self.playlist_result(entries, anime_id, anime_title, anime_description)
+        return self.playlist_result(
+            entries(), anime_id, anime_title, anime_description)

youtube_dlc/extractor/anvato.py

@@ -21,6 +21,16 @@ from ..utils import (
     unsmuggle_url,
 )
 
+# This import causes a ModuleNotFoundError on some systems for unknown reason.
+# See issues:
+# https://github.com/pukkandan/yt-dlp/issues/35
+# https://github.com/ytdl-org/youtube-dl/issues/27449
+# https://github.com/animelover1984/youtube-dl/issues/17
+try:
+    from .anvato_token_generator import NFLTokenGenerator
+except ImportError:
+    NFLTokenGenerator = None
+
 
 def md5_text(s):
     if not isinstance(s, compat_str):
@@ -203,6 +213,10 @@ class AnvatoIE(InfoExtractor):
         'telemundo': 'anvato_mcp_telemundo_web_prod_c5278d51ad46fda4b6ca3d0ea44a7846a054f582'
     }
 
+    _TOKEN_GENERATORS = {
+        'GXvEgwyJeWem8KCYXfeoHWknwP48Mboj': NFLTokenGenerator,
+    }
+
     _API_KEY = '3hwbSuqqT690uxjNYBktSQpa5ZrpYYR0Iofx7NcJHyA'
 
     _ANVP_RE = r'<script[^>]+\bdata-anvp\s*=\s*(["\'])(?P<anvp>(?:(?!\1).)+)\1'
@@ -262,9 +276,12 @@ class AnvatoIE(InfoExtractor):
             'anvrid': anvrid,
             'anvts': server_time,
         }
-        api['anvstk'] = md5_text('%s|%s|%d|%s' % (
-            access_key, anvrid, server_time,
-            self._ANVACK_TABLE.get(access_key, self._API_KEY)))
+        if self._TOKEN_GENERATORS.get(access_key) is not None:
+            api['anvstk2'] = self._TOKEN_GENERATORS[access_key].generate(self, access_key, video_id)
+        else:
+            api['anvstk'] = md5_text('%s|%s|%d|%s' % (
+                access_key, anvrid, server_time,
+                self._ANVACK_TABLE.get(access_key, self._API_KEY)))
 
         return self._download_json(
             video_data_url, video_id, transform_source=strip_jsonp,

youtube_dlc/extractor/aol.py

@@ -3,7 +3,7 @@ from __future__ import unicode_literals
 
 import re
 
-from .common import InfoExtractor
+from .yahoo import YahooIE
 from ..compat import (
     compat_parse_qs,
     compat_urllib_parse_urlparse,
@@ -15,9 +15,9 @@ from ..utils import (
 )
 
 
-class AolIE(InfoExtractor):
+class AolIE(YahooIE):
     IE_NAME = 'aol.com'
-    _VALID_URL = r'(?:aol-video:|https?://(?:www\.)?aol\.(?:com|ca|co\.uk|de|jp)/video/(?:[^/]+/)*)(?P<id>[0-9a-f]+)'
+    _VALID_URL = r'(?:aol-video:|https?://(?:www\.)?aol\.(?:com|ca|co\.uk|de|jp)/video/(?:[^/]+/)*)(?P<id>\d{9}|[0-9a-f]{24}|[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})'
 
     _TESTS = [{
         # video with 5min ID
@@ -76,10 +76,16 @@ class AolIE(InfoExtractor):
     }, {
         'url': 'https://www.aol.jp/video/playlist/5a28e936a1334d000137da0c/5a28f3151e642219fde19831/',
         'only_matching': True,
+    }, {
+        # Yahoo video
+        'url': 'https://www.aol.com/video/play/991e6700-ac02-11ea-99ff-357400036f61/24bbc846-3e30-3c46-915e-fe8ccd7fcc46/',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
+        if '-' in video_id:
+            return self._extract_yahoo_video(video_id, 'us')
 
         response = self._download_json(
             'https://feedapi.b2c.on.aol.com/v1.0/app/videos/aolon/%s/details' % video_id,

youtube_dlc/extractor/ard.py

@@ -226,13 +226,13 @@ class ARDMediathekIE(ARDMediathekBaseIE):
             if doc.tag == 'rss':
                 return GenericIE()._extract_rss(url, video_id, doc)
 
-        title = self._html_search_regex(
+        title = self._og_search_title(webpage, default=None) or self._html_search_regex(
             [r'<h1(?:\s+class="boxTopHeadline")?>(.*?)</h1>',
              r'<meta name="dcterms\.title" content="(.*?)"/>',
              r'<h4 class="headline">(.*?)</h4>',
              r'<title[^>]*>(.*?)</title>'],
             webpage, 'title')
-        description = self._html_search_meta(
+        description = self._og_search_description(webpage, default=None) or self._html_search_meta(
             'dcterms.abstract', webpage, 'description', default=None)
         if description is None:
             description = self._html_search_meta(
@@ -289,18 +289,18 @@ class ARDMediathekIE(ARDMediathekBaseIE):
 
 
 class ARDIE(InfoExtractor):
-    _VALID_URL = r'(?P<mainurl>https?://(www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?P<id>[0-9]+))\.html'
+    _VALID_URL = r'(?P<mainurl>https?://(?:www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?:video-?)?(?P<id>[0-9]+))\.html'
     _TESTS = [{
-        # available till 14.02.2019
-        'url': 'http://www.daserste.de/information/talk/maischberger/videos/das-groko-drama-zerlegen-sich-die-volksparteien-video-102.html',
-        'md5': '8e4ec85f31be7c7fc08a26cdbc5a1f49',
+        # available till 7.01.2022
+        'url': 'https://www.daserste.de/information/talk/maischberger/videos/maischberger-die-woche-video100.html',
+        'md5': '867d8aa39eeaf6d76407c5ad1bb0d4c1',
         'info_dict': {
-            'display_id': 'das-groko-drama-zerlegen-sich-die-volksparteien-video',
-            'id': '102',
+            'display_id': 'maischberger-die-woche',
+            'id': '100',
             'ext': 'mp4',
-            'duration': 4435.0,
-            'title': 'Das GroKo-Drama: Zerlegen sich die Volksparteien?',
-            'upload_date': '20180214',
+            'duration': 3687.0,
+            'title': 'maischberger. die woche vom 7. Januar 2021',
+            'upload_date': '20210107',
             'thumbnail': r're:^https?://.*\.jpg$',
         },
     }, {
@@ -355,17 +355,17 @@ class ARDIE(InfoExtractor):
 class ARDBetaMediathekIE(ARDMediathekBaseIE):
     _VALID_URL = r'https://(?:(?:beta|www)\.)?ardmediathek\.de/(?P<client>[^/]+)/(?P<mode>player|live|video|sendung|sammlung)/(?P<display_id>(?:[^/]+/)*)(?P<video_id>[a-zA-Z0-9]+)'
     _TESTS = [{
-        'url': 'https://ardmediathek.de/ard/video/die-robuste-roswita/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
-        'md5': 'dfdc87d2e7e09d073d5a80770a9ce88f',
+        'url': 'https://www.ardmediathek.de/mdr/video/die-robuste-roswita/Y3JpZDovL21kci5kZS9iZWl0cmFnL2Ntcy84MWMxN2MzZC0wMjkxLTRmMzUtODk4ZS0wYzhlOWQxODE2NGI/',
+        'md5': 'a1dc75a39c61601b980648f7c9f9f71d',
         'info_dict': {
             'display_id': 'die-robuste-roswita',
-            'id': '70153354',
+            'id': '78566716',
             'title': 'Die robuste Roswita',
-            'description': r're:^Der Mord.*trüber ist als die Ilm.',
+            'description': r're:^Der Mord.*totgeglaubte Ehefrau Roswita',
             'duration': 5316,
-            'thumbnail': 'https://img.ardmediathek.de/standard/00/70/15/33/90/-1852531467/16x9/960?mandant=ard',
-            'timestamp': 1577047500,
-            'upload_date': '20191222',
+            'thumbnail': 'https://img.ardmediathek.de/standard/00/78/56/67/84/575672121/16x9/960?mandant=ard',
+            'timestamp': 1596658200,
+            'upload_date': '20200805',
             'ext': 'mp4',
         },
     }, {

youtube_dlc/extractor/audius.py

@@ -0,0 +1,224 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import random
+import re
+
+from .common import InfoExtractor
+from ..utils import ExtractorError, try_get, compat_str, str_or_none
+from ..compat import compat_urllib_parse_unquote
+
+
+class AudiusBaseIE(InfoExtractor):
+    _API_BASE = None
+    _API_V = '/v1'
+
+    def _get_response_data(self, response):
+        if isinstance(response, dict):
+            response_data = response.get('data')
+            if response_data is not None:
+                return response_data
+            if len(response) == 1 and 'message' in response:
+                raise ExtractorError('API error: %s' % response['message'],
+                                     expected=True)
+        raise ExtractorError('Unexpected API response')
+
+    def _select_api_base(self):
+        """Selecting one of the currently available API hosts"""
+        response = super(AudiusBaseIE, self)._download_json(
+            'https://api.audius.co/', None,
+            note='Requesting available API hosts',
+            errnote='Unable to request available API hosts')
+        hosts = self._get_response_data(response)
+        if isinstance(hosts, list):
+            self._API_BASE = random.choice(hosts)
+            return
+        raise ExtractorError('Unable to get available API hosts')
+
+    @staticmethod
+    def _prepare_url(url, title):
+        """
+        Audius removes forward slashes from the uri, but leaves backslashes.
+        The problem is that the current version of Chrome replaces backslashes
+        in the address bar with a forward slashes, so if you copy the link from
+        there and paste it into youtube-dl, you won't be able to download
+        anything from this link, since the Audius API won't be able to resolve
+        this url
+        """
+        url = compat_urllib_parse_unquote(url)
+        title = compat_urllib_parse_unquote(title)
+        if '/' in title or '%2F' in title:
+            fixed_title = title.replace('/', '%5C').replace('%2F', '%5C')
+            return url.replace(title, fixed_title)
+        return url
+
+    def _api_request(self, path, item_id=None, note='Downloading JSON metadata',
+                     errnote='Unable to download JSON metadata',
+                     expected_status=None):
+        if self._API_BASE is None:
+            self._select_api_base()
+        try:
+            response = super(AudiusBaseIE, self)._download_json(
+                '%s%s%s' % (self._API_BASE, self._API_V, path), item_id, note=note,
+                errnote=errnote, expected_status=expected_status)
+        except ExtractorError as exc:
+            # some of Audius API hosts may not work as expected and return HTML
+            if 'Failed to parse JSON' in compat_str(exc):
+                raise ExtractorError('An error occurred while receiving data. Try again',
+                                     expected=True)
+            raise exc
+        return self._get_response_data(response)
+
+    def _resolve_url(self, url, item_id):
+        return self._api_request('/resolve?url=%s' % url, item_id,
+                                 expected_status=404)
+
+
+class AudiusIE(AudiusBaseIE):
+    _VALID_URL = r'''(?x)https?://(?:www\.)?(?:audius\.co/(?P<uploader>[\w\d-]+)(?!/album|/playlist)/(?P<title>\S+))'''
+    _TESTS = [
+        {
+            # URL from Chrome address bar which replace backslash to forward slash
+            'url': 'https://audius.co/test_acc/t%D0%B5%D0%B5%D0%B5est-1.%5E_%7B%7D/%22%3C%3E.%E2%84%96~%60-198631',
+            'md5': '92c35d3e754d5a0f17eef396b0d33582',
+            'info_dict': {
+                'id': 'xd8gY',
+                'title': '''Tеееest/ 1.!@#$%^&*()_+=[]{};'\\\":<>,.?/№~`''',
+                'ext': 'mp3',
+                'description': 'Description',
+                'duration': 30,
+                'track': '''Tеееest/ 1.!@#$%^&*()_+=[]{};'\\\":<>,.?/№~`''',
+                'artist': 'test',
+                'genre': 'Electronic',
+                'thumbnail': r're:https?://.*\.jpg',
+                'view_count': int,
+                'like_count': int,
+                'repost_count': int,
+            }
+        },
+        {
+            # Regular track
+            'url': 'https://audius.co/voltra/radar-103692',
+            'md5': '491898a0a8de39f20c5d6a8a80ab5132',
+            'info_dict': {
+                'id': 'KKdy2',
+                'title': 'RADAR',
+                'ext': 'mp3',
+                'duration': 318,
+                'track': 'RADAR',
+                'artist': 'voltra',
+                'genre': 'Trance',
+                'thumbnail': r're:https?://.*\.jpg',
+                'view_count': int,
+                'like_count': int,
+                'repost_count': int,
+            }
+        },
+    ]
+
+    _ARTWORK_MAP = {
+        "150x150": 150,
+        "480x480": 480,
+        "1000x1000": 1000
+    }
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        uploader, title, track_id = mobj.groups()
+        if track_id is None:
+            url = self._prepare_url(url, title)
+            track_data = self._resolve_url(url, title)
+        else:  # API link
+            track_data = self._api_request('/tracks/%s' % track_id, track_id)
+
+        if not isinstance(track_data, dict):
+            raise ExtractorError('Unexpected API response')
+
+        track_id = track_data.get('id')
+        if track_id is None:
+            raise ExtractorError('Unable to get ID of the track')
+
+        artworks_data = track_data.get('artwork')
+        thumbnails = []
+        if isinstance(artworks_data, dict):
+            for quality_key, thumbnail_url in artworks_data.items():
+                thumbnail = {
+                    "url": thumbnail_url
+                }
+                quality_code = self._ARTWORK_MAP.get(quality_key)
+                if quality_code is not None:
+                    thumbnail['preference'] = quality_code
+                thumbnails.append(thumbnail)
+
+        return {
+            'id': track_id,
+            'title': track_data.get('title', title),
+            'url': '%s/v1/tracks/%s/stream' % (self._API_BASE, track_id),
+            'ext': 'mp3',
+            'description': track_data.get('description'),
+            'duration': track_data.get('duration'),
+            'track': track_data.get('title'),
+            'artist': try_get(track_data, lambda x: x['user']['name'], compat_str),
+            'genre': track_data.get('genre'),
+            'thumbnails': thumbnails,
+            'view_count': track_data.get('play_count'),
+            'like_count': track_data.get('favorite_count'),
+            'repost_count': track_data.get('repost_count'),
+        }
+
+
+class AudiusPlaylistIE(AudiusBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?audius\.co/(?P<uploader>[\w\d-]+)/(?:album|playlist)/(?P<title>\S+)'
+    IE_NAME = 'audius:playlist'
+    _TEST = {
+        'url': 'https://audius.co/test_acc/playlist/test-playlist-22910',
+        'info_dict': {
+            'id': 'DNvjN',
+            'title': 'test playlist',
+            'description': 'Test description\n\nlol',
+        },
+        'playlist_count': 175,
+    }
+
+    def _build_playlist(self, tracks):
+        entries = []
+        for track in tracks:
+            if not isinstance(track, dict):
+                raise ExtractorError('Unexpected API response')
+            track_id = str_or_none(track.get('id'))
+            if not track_id:
+                raise ExtractorError('Unable to get track ID from playlist')
+            entries.append(self.url_result(
+                '%s%s/tracks/%s' % (self._API_BASE, self._API_V, track_id),
+                ie=AudiusIE.ie_key(), video_id=track_id))
+        return entries
+
+    def _real_extract(self, url):
+        self._select_api_base()
+        mobj = re.match(self._VALID_URL, url)
+        uploader, title = mobj.groups()
+        url = self._prepare_url(url, title)
+        playlist_response = self._resolve_url(url, title)
+
+        if not isinstance(playlist_response, list) or len(playlist_response) != 1:
+            raise ExtractorError('Unexpected API response')
+
+        playlist_data = playlist_response[0]
+        if not isinstance(playlist_data, dict):
+            raise ExtractorError('Unexpected API response')
+
+        playlist_id = playlist_data.get('id')
+        if playlist_id is None:
+            raise ExtractorError('Unable to get playlist ID')
+
+        playlist_tracks = self._api_request(
+            '/playlists/%s/tracks' % playlist_id,
+            title, note='Downloading playlist tracks metadata',
+            errnote='Unable to download playlist tracks metadata')
+        if not isinstance(playlist_tracks, list):
+            raise ExtractorError('Unexpected API response')
+
+        entries = self._build_playlist(playlist_tracks)
+        return self.playlist_result(entries, playlist_id,
+                                    playlist_data.get('playlist_name', title),
+                                    playlist_data.get('description'))

youtube_dlc/extractor/bilibili.py

@@ -2,9 +2,10 @@
 from __future__ import unicode_literals
 
 import hashlib
+import json
 import re
 
-from .common import InfoExtractor
+from .common import InfoExtractor, SearchInfoExtractor
 from ..compat import (
     compat_parse_qs,
     compat_urlparse,
@@ -32,13 +33,14 @@ class BiliBiliIE(InfoExtractor):
                             (?:
                                 video/[aA][vV]|
                                 anime/(?P<anime_id>\d+)/play\#
-                            )(?P<id_bv>\d+)|
-                            video/[bB][vV](?P<id>[^/?#&]+)
+                            )(?P<id>\d+)|
+                            video/[bB][vV](?P<id_bv>[^/?#&]+)
                         )
+                        (?:/?\?p=(?P<page>\d+))?
                     '''
 
     _TESTS = [{
-        'url': 'http://www.bilibili.tv/video/av1074402/',
+        'url': 'http://www.bilibili.com/video/av1074402/',
         'md5': '5f7d29e1a2872f3df0cf76b1f87d3788',
         'info_dict': {
             'id': '1074402',
@@ -56,6 +58,10 @@ class BiliBiliIE(InfoExtractor):
         # Tested in BiliBiliBangumiIE
         'url': 'http://bangumi.bilibili.com/anime/1869/play#40062',
         'only_matching': True,
+    }, {
+        # bilibili.tv
+        'url': 'http://www.bilibili.tv/video/av1074402/',
+        'only_matching': True,
     }, {
         'url': 'http://bangumi.bilibili.com/anime/5802/play#100643',
         'md5': '3f721ad1e75030cc06faf73587cfec57',
@@ -124,12 +130,20 @@ class BiliBiliIE(InfoExtractor):
         url, smuggled_data = unsmuggle_url(url, {})
 
         mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id') or mobj.group('id_bv')
+        video_id = mobj.group('id_bv') or mobj.group('id')
+
+        av_id, bv_id = self._get_video_id_set(video_id, mobj.group('id_bv') is not None)
+        video_id = av_id
+
         anime_id = mobj.group('anime_id')
+        page_id = mobj.group('page')
         webpage = self._download_webpage(url, video_id)
 
         if 'anime/' not in url:
             cid = self._search_regex(
+                r'\bcid(?:["\']:|=)(\d+),["\']page(?:["\']:|=)' + str(page_id), webpage, 'cid',
+                default=None
+            ) or self._search_regex(
                 r'\bcid(?:["\']:|=)(\d+)', webpage, 'cid',
                 default=None
             ) or compat_parse_qs(self._search_regex(
@@ -207,9 +221,9 @@ class BiliBiliIE(InfoExtractor):
             break
 
         title = self._html_search_regex(
-            ('<h1[^>]+\btitle=(["\'])(?P<title>(?:(?!\1).)+)\1',
-             '(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
-            group='title')
+            (r'<h1[^>]+\btitle=(["\'])(?P<title>(?:(?!\1).)+)\1',
+             r'(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
+            group='title') + ('_p' + str(page_id) if page_id is not None else '')
         description = self._html_search_meta('description', webpage)
         timestamp = unified_timestamp(self._html_search_regex(
             r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time',
@@ -219,7 +233,8 @@ class BiliBiliIE(InfoExtractor):
 
         # TODO 'view_count' requires deobfuscating Javascript
         info = {
-            'id': video_id,
+            'id': str(video_id) if page_id is None else '%s_p%s' % (video_id, page_id),
+            'cid': cid,
             'title': title,
             'description': description,
             'timestamp': timestamp,
@@ -235,27 +250,134 @@ class BiliBiliIE(InfoExtractor):
                 'uploader': uploader_mobj.group('name'),
                 'uploader_id': uploader_mobj.group('id'),
             })
+
         if not info.get('uploader'):
             info['uploader'] = self._html_search_meta(
                 'author', webpage, 'uploader', default=None)
 
+        comments = None
+        if self._downloader.params.get('getcomments', False):
+            comments = self._get_all_comment_pages(video_id)
+
+        raw_danmaku = self._get_raw_danmaku(video_id, cid)
+
+        raw_tags = self._get_tags(video_id)
+        tags = list(map(lambda x: x['tag_name'], raw_tags))
+
+        top_level_info = {
+            'raw_danmaku': raw_danmaku,
+            'comments': comments,
+            'comment_count': len(comments) if comments is not None else None,
+            'tags': tags,
+            'raw_tags': raw_tags,
+        }
+
+        '''
+        # Requires https://github.com/m13253/danmaku2ass which is licenced under GPL3
+        # See https://github.com/animelover1984/youtube-dl
+        danmaku = NiconicoIE.CreateDanmaku(raw_danmaku, commentType='Bilibili', x=1024, y=576)
+        entries[0]['subtitles'] = {
+            'danmaku': [{
+                'ext': 'ass',
+                'data': danmaku
+            }]
+        }
+        '''
+
         for entry in entries:
             entry.update(info)
 
         if len(entries) == 1:
+            entries[0].update(top_level_info)
             return entries[0]
         else:
             for idx, entry in enumerate(entries):
                 entry['id'] = '%s_part%d' % (video_id, (idx + 1))
 
-            return {
+            global_info = {
                 '_type': 'multi_video',
                 'id': video_id,
+                'bv_id': bv_id,
                 'title': title,
                 'description': description,
                 'entries': entries,
             }
 
+            global_info.update(info)
+            global_info.update(top_level_info)
+
+            return global_info
+
+    def _get_video_id_set(self, id, is_bv):
+        query = {'bvid': id} if is_bv else {'aid': id}
+        response = self._download_json(
+            "http://api.bilibili.cn/x/web-interface/view",
+            id, query=query,
+            note='Grabbing original ID via API')
+
+        if response['code'] == -400:
+            raise ExtractorError('Video ID does not exist', expected=True, video_id=id)
+        elif response['code'] != 0:
+            raise ExtractorError('Unknown error occurred during API check (code %s)' % response['code'], expected=True, video_id=id)
+        return (response['data']['aid'], response['data']['bvid'])
+
+    # recursive solution to getting every page of comments for the video
+    # we can stop when we reach a page without any comments
+    def _get_all_comment_pages(self, video_id, commentPageNumber=0):
+        comment_url = "https://api.bilibili.com/x/v2/reply?jsonp=jsonp&pn=%s&type=1&oid=%s&sort=2&_=1567227301685" % (commentPageNumber, video_id)
+        json_str = self._download_webpage(
+            comment_url, video_id,
+            note='Extracting comments from page %s' % (commentPageNumber))
+        replies = json.loads(json_str)['data']['replies']
+        if replies is None:
+            return []
+        return self._get_all_children(replies) + self._get_all_comment_pages(video_id, commentPageNumber + 1)
+
+    # extracts all comments in the tree
+    def _get_all_children(self, replies):
+        if replies is None:
+            return []
+
+        ret = []
+        for reply in replies:
+            author = reply['member']['uname']
+            author_id = reply['member']['mid']
+            id = reply['rpid']
+            text = reply['content']['message']
+            timestamp = reply['ctime']
+            parent = reply['parent'] if reply['parent'] != 0 else 'root'
+
+            comment = {
+                "author": author,
+                "author_id": author_id,
+                "id": id,
+                "text": text,
+                "timestamp": timestamp,
+                "parent": parent,
+            }
+            ret.append(comment)
+
+            # from the JSON, the comment structure seems arbitrarily deep, but I could be wrong.
+            # Regardless, this should work.
+            ret += self._get_all_children(reply['replies'])
+
+        return ret
+
+    def _get_raw_danmaku(self, video_id, cid):
+        # This will be useful if I decide to scrape all pages instead of doing them individually
+        # cid_url = "https://www.bilibili.com/widget/getPageList?aid=%s" % (video_id)
+        # cid_str = self._download_webpage(cid_url, video_id, note=False)
+        # cid = json.loads(cid_str)[0]['cid']
+
+        danmaku_url = "https://comment.bilibili.com/%s.xml" % (cid)
+        danmaku = self._download_webpage(danmaku_url, video_id, note='Downloading danmaku comments')
+        return danmaku
+
+    def _get_tags(self, video_id):
+        tags_url = "https://api.bilibili.com/x/tag/archive/tags?aid=%s" % (video_id)
+        tags_json = self._download_json(tags_url, video_id, note='Downloading tags')
+        return tags_json['data']
+
 
 class BiliBiliBangumiIE(InfoExtractor):
     _VALID_URL = r'https?://bangumi\.bilibili\.com/anime/(?P<id>\d+)'
@@ -324,6 +446,73 @@ class BiliBiliBangumiIE(InfoExtractor):
             season_info.get('bangumi_title'), season_info.get('evaluate'))
 
 
+class BilibiliChannelIE(InfoExtractor):
+    _VALID_URL = r'https?://space.bilibili\.com/(?P<id>\d+)'
+    # May need to add support for pagination? Need to find a user with many video uploads to test
+    _API_URL = "https://api.bilibili.com/x/space/arc/search?mid=%s&pn=1&ps=25&jsonp=jsonp"
+    _TEST = {}  # TODO: Add tests
+
+    def _real_extract(self, url):
+        list_id = self._match_id(url)
+        json_str = self._download_webpage(self._API_URL % list_id, "None")
+
+        json_parsed = json.loads(json_str)
+        entries = [{
+            '_type': 'url',
+            'ie_key': BiliBiliIE.ie_key(),
+            'url': ('https://www.bilibili.com/video/%s' %
+                    entry['bvid']),
+            'id': entry['bvid'],
+        } for entry in json_parsed['data']['list']['vlist']]
+
+        return {
+            '_type': 'playlist',
+            'id': list_id,
+            'entries': entries
+        }
+
+
+class BiliBiliSearchIE(SearchInfoExtractor):
+    IE_DESC = 'Bilibili video search, "bilisearch" keyword'
+    _MAX_RESULTS = 100000
+    _SEARCH_KEY = 'bilisearch'
+    MAX_NUMBER_OF_RESULTS = 1000
+
+    def _get_n_results(self, query, n):
+        """Get a specified number of results for a query"""
+
+        entries = []
+        pageNumber = 0
+        while True:
+            pageNumber += 1
+            # FIXME
+            api_url = "https://api.bilibili.com/x/web-interface/search/type?context=&page=%s&order=pubdate&keyword=%s&duration=0&tids_2=&__refresh__=true&search_type=video&tids=0&highlight=1" % (pageNumber, query)
+            json_str = self._download_webpage(
+                api_url, "None", query={"Search_key": query},
+                note='Extracting results from page %s' % pageNumber)
+            data = json.loads(json_str)['data']
+
+            # FIXME: this is hideous
+            if "result" not in data:
+                return {
+                    '_type': 'playlist',
+                    'id': query,
+                    'entries': entries[:n]
+                }
+
+            videos = data['result']
+            for video in videos:
+                e = self.url_result(video['arcurl'], 'BiliBili', str(video['aid']))
+                entries.append(e)
+
+            if(len(entries) >= n or len(videos) >= BiliBiliSearchIE.MAX_NUMBER_OF_RESULTS):
+                return {
+                    '_type': 'playlist',
+                    'id': query,
+                    'entries': entries[:n]
+                }
+
+
 class BilibiliAudioBaseIE(InfoExtractor):
     def _call_api(self, path, sid, query=None):
         if not query:

youtube_dlc/extractor/cbs.py

@@ -59,7 +59,7 @@ class CBSIE(CBSBaseIE):
             'http://can.cbs.com/thunder/player/videoPlayerService.php',
             content_id, query={'partner': site, 'contentId': content_id})
         video_data = xpath_element(items_data, './/item')
-        title = xpath_text(video_data, 'videoTitle', 'title', True)
+        title = xpath_text(video_data, 'videoTitle', 'title') or xpath_text(video_data, 'videotitle', 'title')
         tp_path = 'dJ5BDC/media/guid/%d/%s' % (mpx_acc, content_id)
         tp_release_url = 'http://link.theplatform.com/s/' + tp_path
 

youtube_dlc/extractor/comedycentral.py

@@ -1,142 +1,51 @@
 from __future__ import unicode_literals
 
 from .mtv import MTVServicesInfoExtractor
-from .common import InfoExtractor
 
 
 class ComedyCentralIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'''(?x)https?://(?:www\.)?cc\.com/
-        (video-clips|episodes|cc-studios|video-collections|shows(?=/[^/]+/(?!full-episodes)))
-        /(?P<title>.*)'''
+    _VALID_URL = r'https?://(?:www\.)?cc\.com/(?:episodes|video(?:-clips)?)/(?P<id>[0-9a-z]{6})'
     _FEED_URL = 'http://comedycentral.com/feeds/mrss/'
 
     _TESTS = [{
-        'url': 'http://www.cc.com/video-clips/kllhuv/stand-up-greg-fitzsimmons--uncensored---too-good-of-a-mother',
-        'md5': 'c4f48e9eda1b16dd10add0744344b6d8',
+        'url': 'http://www.cc.com/video-clips/5ke9v2/the-daily-show-with-trevor-noah-doc-rivers-and-steve-ballmer---the-nba-player-strike',
+        'md5': 'b8acb347177c680ff18a292aa2166f80',
         'info_dict': {
-            'id': 'cef0cbb3-e776-4bc9-b62e-8016deccb354',
+            'id': '89ccc86e-1b02-4f83-b0c9-1d9592ecd025',
             'ext': 'mp4',
-            'title': 'CC:Stand-Up|August 18, 2013|1|0101|Uncensored - Too Good of a Mother',
-            'description': 'After a certain point, breastfeeding becomes c**kblocking.',
-            'timestamp': 1376798400,
-            'upload_date': '20130818',
+            'title': 'The Daily Show with Trevor Noah|August 28, 2020|25|25149|Doc Rivers and Steve Ballmer - The NBA Player Strike',
+            'description': 'md5:5334307c433892b85f4f5e5ac9ef7498',
+            'timestamp': 1598670000,
+            'upload_date': '20200829',
         },
     }, {
-        'url': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/interviews/6yx39d/exclusive-rand-paul-extended-interview',
+        'url': 'http://www.cc.com/episodes/pnzzci/drawn-together--american-idol--parody-clip-show-season-3-ep-314',
         'only_matching': True,
-    }]
-
-
-class ComedyCentralFullEpisodesIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'''(?x)https?://(?:www\.)?cc\.com/
-        (?:full-episodes|shows(?=/[^/]+/full-episodes))
-        /(?P<id>[^?]+)'''
-    _FEED_URL = 'http://comedycentral.com/feeds/mrss/'
-
-    _TESTS = [{
-        'url': 'http://www.cc.com/full-episodes/pv391a/the-daily-show-with-trevor-noah-november-28--2016---ryan-speedo-green-season-22-ep-22028',
-        'info_dict': {
-            'description': 'Donald Trump is accused of exploiting his president-elect status for personal gain, Cuban leader Fidel Castro dies, and Ryan Speedo Green discusses "Sing for Your Life."',
-            'title': 'November 28, 2016 - Ryan Speedo Green',
-        },
-        'playlist_count': 4,
     }, {
-        'url': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        playlist_id = self._match_id(url)
-        webpage = self._download_webpage(url, playlist_id)
-        mgid = self._extract_mgid(webpage, url, data_zone='t2_lc_promo1')
-        videos_info = self._get_videos_info(mgid)
-        return videos_info
-
-
-class ToshIE(MTVServicesInfoExtractor):
-    IE_DESC = 'Tosh.0'
-    _VALID_URL = r'^https?://tosh\.cc\.com/video-(?:clips|collections)/[^/]+/(?P<videotitle>[^/?#]+)'
-    _FEED_URL = 'http://tosh.cc.com/feeds/mrss'
-
-    _TESTS = [{
-        'url': 'http://tosh.cc.com/video-clips/68g93d/twitter-users-share-summer-plans',
-        'info_dict': {
-            'description': 'Tosh asked fans to share their summer plans.',
-            'title': 'Twitter Users Share Summer Plans',
-        },
-        'playlist': [{
-            'md5': 'f269e88114c1805bb6d7653fecea9e06',
-            'info_dict': {
-                'id': '90498ec2-ed00-11e0-aca6-0026b9414f30',
-                'ext': 'mp4',
-                'title': 'Tosh.0|June 9, 2077|2|211|Twitter Users Share Summer Plans',
-                'description': 'Tosh asked fans to share their summer plans.',
-                'thumbnail': r're:^https?://.*\.jpg',
-                # It's really reported to be published on year 2077
-                'upload_date': '20770610',
-                'timestamp': 3390510600,
-                'subtitles': {
-                    'en': 'mincount:3',
-                },
-            },
-        }]
-    }, {
-        'url': 'http://tosh.cc.com/video-collections/x2iz7k/just-plain-foul/m5q4fp',
+        'url': 'https://www.cc.com/video/k3sdvm/the-daily-show-with-jon-stewart-exclusive-the-fourth-estate',
         'only_matching': True,
     }]
 
 
 class ComedyCentralTVIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?comedycentral\.tv/(?:staffeln|shows)/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?comedycentral\.tv/folgen/(?P<id>[0-9a-z]{6})'
     _TESTS = [{
-        'url': 'http://www.comedycentral.tv/staffeln/7436-the-mindy-project-staffel-4',
+        'url': 'https://www.comedycentral.tv/folgen/pxdpec/josh-investigates-klimawandel-staffel-1-ep-1',
         'info_dict': {
-            'id': 'local_playlist-f99b626bdfe13568579a',
-            'ext': 'flv',
-            'title': 'Episode_the-mindy-project_shows_season-4_episode-3_full-episode_part1',
+            'id': '15907dc3-ec3c-11e8-a442-0e40cf2fc285',
+            'ext': 'mp4',
+            'title': 'Josh Investigates',
+            'description': 'Steht uns das Ende der Welt bevor?',
         },
-        'params': {
-            # rtmp download
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://www.comedycentral.tv/shows/1074-workaholics',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.comedycentral.tv/shows/1727-the-mindy-project/bonus',
-        'only_matching': True,
     }]
+    _FEED_URL = 'http://feeds.mtvnservices.com/od/feed/intl-mrss-player-feed'
+    _GEO_COUNTRIES = ['DE']
 
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, video_id)
-
-        mrss_url = self._search_regex(
-            r'data-mrss=(["\'])(?P<url>(?:(?!\1).)+)\1',
-            webpage, 'mrss url', group='url')
-
-        return self._get_videos_info_from_url(mrss_url, video_id)
-
-
-class ComedyCentralShortnameIE(InfoExtractor):
-    _VALID_URL = r'^:(?P<id>tds|thedailyshow|theopposition)$'
-    _TESTS = [{
-        'url': ':tds',
-        'only_matching': True,
-    }, {
-        'url': ':thedailyshow',
-        'only_matching': True,
-    }, {
-        'url': ':theopposition',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        shortcut_map = {
-            'tds': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-            'thedailyshow': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-            'theopposition': 'http://www.cc.com/shows/the-opposition-with-jordan-klepper/full-episodes',
+    def _get_feed_query(self, uri):
+        return {
+            'accountOverride': 'intl.mtvi.com',
+            'arcEp': 'web.cc.tv',
+            'ep': 'b9032c3a',
+            'imageEp': 'web.cc.tv',
+            'mgid': uri,
         }
-        return self.url_result(shortcut_map[video_id])

youtube_dlc/extractor/common.py

@@ -1375,8 +1375,8 @@ class InfoExtractor(object):
                        'order': ['vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
             'acodec': {'type': 'ordered', 'regex': True,
                        'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
-            'protocol': {'type': 'ordered', 'regex': True,
-                         'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
+            'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
+                      'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
             'vext': {'type': 'ordered', 'field': 'video_ext',
                      'order': ('mp4', 'webm', 'flv', '', 'none'),
                      'order_free': ('webm', 'mp4', 'flv', '', 'none')},
@@ -1384,14 +1384,14 @@ class InfoExtractor(object):
                      'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
                      'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
             'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
-            'extractor_preference': {'priority': True, 'type': 'extractor'},
-            'has_video': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'has_audio': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'language_preference': {'priority': True, 'convert': 'ignore'},
+            'ie_pref': {'priority': True, 'type': 'extractor', 'field': 'extractor_preference'},
+            'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
+            'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
+            'lang': {'priority': True, 'convert': 'ignore', 'field': 'language_preference'},
             'quality': {'priority': True, 'convert': 'float_none'},
             'filesize': {'convert': 'bytes'},
-            'filesize_approx': {'convert': 'bytes'},
-            'format_id': {'convert': 'string'},
+            'fs_approx': {'convert': 'bytes', 'field': 'filesize_approx'},
+            'id': {'convert': 'string', 'field': 'format_id'},
             'height': {'convert': 'float_none'},
             'width': {'convert': 'float_none'},
             'fps': {'convert': 'float_none'},
@@ -1399,32 +1399,42 @@ class InfoExtractor(object):
             'vbr': {'convert': 'float_none'},
             'abr': {'convert': 'float_none'},
             'asr': {'convert': 'float_none'},
-            'source_preference': {'convert': 'ignore'},
+            'source': {'convert': 'ignore', 'field': 'source_preference'},
+
             'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
-            'bitrate': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
-            'filesize_estimate': {'type': 'combined', 'same_limit': True, 'field': ('filesize', 'filesize_approx')},
-            'extension': {'type': 'combined', 'field': ('vext', 'aext')},
-            'dimension': {'type': 'multiple', 'field': ('height', 'width'), 'function': min},  # not named as 'resolution' because such a field exists
-            'res': {'type': 'alias', 'field': 'dimension'},
-            'ext': {'type': 'alias', 'field': 'extension'},
-            'br': {'type': 'alias', 'field': 'bitrate'},
+            'br': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
+            'size': {'type': 'combined', 'same_limit': True, 'field': ('filesize', 'fs_approx')},
+            'ext': {'type': 'combined', 'field': ('vext', 'aext')},
+            'res': {'type': 'multiple', 'field': ('height', 'width'), 'function': min},
+
+            # Most of these exist only for compatibility reasons
+            'dimension': {'type': 'alias', 'field': 'res'},
+            'resolution': {'type': 'alias', 'field': 'res'},
+            'extension': {'type': 'alias', 'field': 'ext'},
+            'bitrate': {'type': 'alias', 'field': 'br'},
             'total_bitrate': {'type': 'alias', 'field': 'tbr'},
             'video_bitrate': {'type': 'alias', 'field': 'vbr'},
             'audio_bitrate': {'type': 'alias', 'field': 'abr'},
             'framerate': {'type': 'alias', 'field': 'fps'},
-            'lang': {'type': 'alias', 'field': 'language_preference'},  # not named as 'language' because such a field exists
-            'proto': {'type': 'alias', 'field': 'protocol'},
-            'source': {'type': 'alias', 'field': 'source_preference'},
-            'size': {'type': 'alias', 'field': 'filesize_estimate'},
+            'language_preference': {'type': 'alias', 'field': 'lang'},  # not named as 'language' because such a field exists
+            'protocol': {'type': 'alias', 'field': 'proto'},
+            'source_preference': {'type': 'alias', 'field': 'source'},
+            'filesize_approx': {'type': 'alias', 'field': 'fs_approx'},
+            'filesize_estimate': {'type': 'alias', 'field': 'size'},
             'samplerate': {'type': 'alias', 'field': 'asr'},
             'video_ext': {'type': 'alias', 'field': 'vext'},
             'audio_ext': {'type': 'alias', 'field': 'aext'},
             'video_codec': {'type': 'alias', 'field': 'vcodec'},
             'audio_codec': {'type': 'alias', 'field': 'acodec'},
-            'video': {'type': 'alias', 'field': 'has_video'},
-            'audio': {'type': 'alias', 'field': 'has_audio'},
-            'extractor': {'type': 'alias', 'field': 'extractor_preference'},
-            'preference': {'type': 'alias', 'field': 'extractor_preference'}}
+            'video': {'type': 'alias', 'field': 'hasvid'},
+            'has_video': {'type': 'alias', 'field': 'hasvid'},
+            'audio': {'type': 'alias', 'field': 'hasaud'},
+            'has_audio': {'type': 'alias', 'field': 'hasaud'},
+            'extractor': {'type': 'alias', 'field': 'ie_pref'},
+            'preference': {'type': 'alias', 'field': 'ie_pref'},
+            'extractor_preference': {'type': 'alias', 'field': 'ie_pref'},
+            'format_id': {'type': 'alias', 'field': 'id'},
+        }
 
         _order = []
 

youtube_dlc/extractor/cspan.py

@@ -8,11 +8,14 @@ from ..utils import (
     ExtractorError,
     extract_attributes,
     find_xpath_attr,
+    get_element_by_attribute,
     get_element_by_class,
     int_or_none,
     js_to_json,
     merge_dicts,
+    parse_iso8601,
     smuggle_url,
+    str_to_int,
     unescapeHTML,
 )
 from .senateisvp import SenateISVPIE
@@ -116,8 +119,30 @@ class CSpanIE(InfoExtractor):
                 jwsetup, video_id, require_title=False, m3u8_id='hls',
                 base_url=url)
             add_referer(info['formats'])
+            for subtitles in info['subtitles'].values():
+                for subtitle in subtitles:
+                    ext = determine_ext(subtitle['url'])
+                    if ext == 'php':
+                        ext = 'vtt'
+                    subtitle['ext'] = ext
             ld_info = self._search_json_ld(webpage, video_id, default={})
-            return merge_dicts(info, ld_info)
+            title = get_element_by_class('video-page-title', webpage) or \
+                self._og_search_title(webpage)
+            description = get_element_by_attribute('itemprop', 'description', webpage) or \
+                self._html_search_meta(['og:description', 'description'], webpage)
+            return merge_dicts(info, ld_info, {
+                'title': title,
+                'thumbnail': get_element_by_attribute('itemprop', 'thumbnailUrl', webpage),
+                'description': description,
+                'timestamp': parse_iso8601(get_element_by_attribute('itemprop', 'uploadDate', webpage)),
+                'location': get_element_by_attribute('itemprop', 'contentLocation', webpage),
+                'duration': int_or_none(self._search_regex(
+                    r'jwsetup\.seclength\s*=\s*(\d+);',
+                    webpage, 'duration', fatal=False)),
+                'view_count': str_to_int(self._search_regex(
+                    r"<span[^>]+class='views'[^>]*>([\d,]+)\s+Views</span>",
+                    webpage, 'views', fatal=False)),
+            })
 
         # Obsolete
         # We first look for clipid, because clipprog always appears before

youtube_dlc/extractor/extractors.py

@@ -50,7 +50,10 @@ from .animelab import (
     AnimeLabIE,
     AnimeLabShowsIE,
 )
-from .americastestkitchen import AmericasTestKitchenIE
+from .americastestkitchen import (
+    AmericasTestKitchenIE,
+    AmericasTestKitchenSeasonIE,
+)
 from .animeondemand import AnimeOnDemandIE
 from .anvato import AnvatoIE
 from .aol import AolIE
@@ -87,6 +90,7 @@ from .atvat import ATVAtIE
 from .audimedia import AudiMediaIE
 from .audioboom import AudioBoomIE
 from .audiomack import AudiomackIE, AudiomackAlbumIE
+from .audius import AudiusIE, AudiusPlaylistIE
 from .awaan import (
     AWAANIE,
     AWAANVideoIE,
@@ -119,10 +123,12 @@ from .bigflix import BigflixIE
 from .bild import BildIE
 from .bilibili import (
     BiliBiliIE,
+    BiliBiliSearchIE,
     BiliBiliBangumiIE,
     BilibiliAudioIE,
     BilibiliAudioAlbumIE,
     BiliBiliPlayerIE,
+    BilibiliChannelIE,
 )
 from .biobiochiletv import BioBioChileTVIE
 from .bitchute import (
@@ -244,11 +250,8 @@ from .cnn import (
 )
 from .coub import CoubIE
 from .comedycentral import (
-    ComedyCentralFullEpisodesIE,
     ComedyCentralIE,
-    ComedyCentralShortnameIE,
     ComedyCentralTVIE,
-    ToshIE,
 )
 from .commonmistakes import CommonMistakesIE, UnicodeBOMIE
 from .commonprotocols import (
@@ -551,7 +554,10 @@ from .karaoketv import KaraoketvIE
 from .karrierevideos import KarriereVideosIE
 from .keezmovies import KeezMoviesIE
 from .ketnet import KetnetIE
-from .khanacademy import KhanAcademyIE
+from .khanacademy import (
+    KhanAcademyIE,
+    KhanAcademyUnitIE,
+)
 from .kickstarter import KickStarterIE
 from .kinja import KinjaEmbedIE
 from .kinopoisk import KinoPoiskIE
@@ -674,6 +680,16 @@ from .microsoftvirtualacademy import (
     MicrosoftVirtualAcademyIE,
     MicrosoftVirtualAcademyCourseIE,
 )
+from .mildom import (
+    MildomIE,
+    MildomVodIE,
+    MildomUserVodIE,
+)
+from .minds import (
+    MindsIE,
+    MindsChannelIE,
+    MindsGroupIE,
+)
 from .ministrygrid import MinistryGridIE
 from .minoto import MinotoIE
 from .miomio import MioMioIE
@@ -1154,6 +1170,10 @@ from .stitcher import StitcherIE
 from .sport5 import Sport5IE
 from .sportbox import SportBoxIE
 from .sportdeutschland import SportDeutschlandIE
+from .spotify import (
+    SpotifyIE,
+    SpotifyShowIE,
+)
 from .spreaker import (
     SpreakerIE,
     SpreakerPageIE,
@@ -1262,6 +1282,10 @@ from .toutv import TouTvIE
 from .toypics import ToypicsUserIE, ToypicsIE
 from .traileraddict import TrailerAddictIE
 from .trilulilu import TriluliluIE
+from .trovo import (
+    TrovoIE,
+    TrovoVodIE,
+)
 from .trunews import TruNewsIE
 from .trutv import TruTVIE
 from .tube8 import Tube8IE

youtube_dlc/extractor/franceculture.py

@@ -11,7 +11,7 @@ from ..utils import (
 
 class FranceCultureIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?franceculture\.fr/emissions/(?:[^/]+/)*(?P<id>[^/?#&]+)'
-    _TEST = {
+    _TESTS = [{
         'url': 'http://www.franceculture.fr/emissions/carnet-nomade/rendez-vous-au-pays-des-geeks',
         'info_dict': {
             'id': 'rendez-vous-au-pays-des-geeks',
@@ -20,10 +20,14 @@ class FranceCultureIE(InfoExtractor):
             'title': 'Rendez-vous au pays des geeks',
             'thumbnail': r're:^https?://.*\.jpg$',
             'upload_date': '20140301',
-            'timestamp': 1393642916,
+            'timestamp': 1393700400,
             'vcodec': 'none',
         }
-    }
+    }, {
+        # no thumbnail
+        'url': 'https://www.franceculture.fr/emissions/la-recherche-montre-en-main/la-recherche-montre-en-main-du-mercredi-10-octobre-2018',
+        'only_matching': True,
+    }]
 
     def _real_extract(self, url):
         display_id = self._match_id(url)
@@ -36,19 +40,19 @@ class FranceCultureIE(InfoExtractor):
                     </h1>|
                     <div[^>]+class="[^"]*?(?:title-zone-diffusion|heading-zone-(?:wrapper|player-button))[^"]*?"[^>]*>
                 ).*?
-                (<button[^>]+data-asset-source="[^"]+"[^>]+>)
+                (<button[^>]+data-(?:url|asset-source)="[^"]+"[^>]+>)
             ''',
             webpage, 'video data'))
 
-        video_url = video_data['data-asset-source']
-        title = video_data.get('data-asset-title') or self._og_search_title(webpage)
+        video_url = video_data.get('data-url') or video_data['data-asset-source']
+        title = video_data.get('data-asset-title') or video_data.get('data-diffusion-title') or self._og_search_title(webpage)
 
         description = self._html_search_regex(
             r'(?s)<div[^>]+class="intro"[^>]*>.*?<h2>(.+?)</h2>',
             webpage, 'description', default=None)
         thumbnail = self._search_regex(
             r'(?s)<figure[^>]+itemtype="https://schema.org/ImageObject"[^>]*>.*?<img[^>]+(?:data-dejavu-)?src="([^"]+)"',
-            webpage, 'thumbnail', fatal=False)
+            webpage, 'thumbnail', default=None)
         uploader = self._html_search_regex(
             r'(?s)<span class="author">(.*?)</span>',
             webpage, 'uploader', default=None)
@@ -64,6 +68,6 @@ class FranceCultureIE(InfoExtractor):
             'ext': ext,
             'vcodec': 'none' if ext == 'mp3' else None,
             'uploader': uploader,
-            'timestamp': int_or_none(video_data.get('data-asset-created-date')),
+            'timestamp': int_or_none(video_data.get('data-start-time')) or int_or_none(video_data.get('data-asset-created-date')),
             'duration': int_or_none(video_data.get('data-duration')),
         }

youtube_dlc/extractor/khanacademy.py

@@ -1,82 +1,107 @@
 from __future__ import unicode_literals
 
-import re
+import json
 
 from .common import InfoExtractor
 from ..utils import (
-    unified_strdate,
+    int_or_none,
+    parse_iso8601,
+    try_get,
 )
 
 
-class KhanAcademyIE(InfoExtractor):
-    _VALID_URL = r'^https?://(?:(?:www|api)\.)?khanacademy\.org/(?P<key>[^/]+)/(?:[^/]+/){,2}(?P<id>[^?#/]+)(?:$|[?#])'
-    IE_NAME = 'KhanAcademy'
+class KhanAcademyBaseIE(InfoExtractor):
+    _VALID_URL_TEMPL = r'https?://(?:www\.)?khanacademy\.org/(?P<id>(?:[^/]+/){%s}%s[^?#/&]+)'
 
-    _TESTS = [{
-        'url': 'http://www.khanacademy.org/video/one-time-pad',
-        'md5': '7b391cce85e758fb94f763ddc1bbb979',
+    def _parse_video(self, video):
+        return {
+            '_type': 'url_transparent',
+            'url': video['youtubeId'],
+            'id': video.get('slug'),
+            'title': video.get('title'),
+            'thumbnail': video.get('imageUrl') or video.get('thumbnailUrl'),
+            'duration': int_or_none(video.get('duration')),
+            'description': video.get('description'),
+            'ie_key': 'Youtube',
+        }
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        component_props = self._parse_json(self._download_json(
+            'https://www.khanacademy.org/api/internal/graphql',
+            display_id, query={
+                'hash': 1604303425,
+                'variables': json.dumps({
+                    'path': display_id,
+                    'queryParams': '',
+                }),
+            })['data']['contentJson'], display_id)['componentProps']
+        return self._parse_component_props(component_props)
+
+
+class KhanAcademyIE(KhanAcademyBaseIE):
+    IE_NAME = 'khanacademy'
+    _VALID_URL = KhanAcademyBaseIE._VALID_URL_TEMPL % ('4', 'v/')
+    _TEST = {
+        'url': 'https://www.khanacademy.org/computing/computer-science/cryptography/crypt/v/one-time-pad',
+        'md5': '9c84b7b06f9ebb80d22a5c8dedefb9a0',
         'info_dict': {
-            'id': 'one-time-pad',
-            'ext': 'webm',
+            'id': 'FlIG3TvQCBQ',
+            'ext': 'mp4',
             'title': 'The one-time pad',
             'description': 'The perfect cipher',
             'duration': 176,
             'uploader': 'Brit Cruise',
             'uploader_id': 'khanacademy',
             'upload_date': '20120411',
+            'timestamp': 1334170113,
+            'license': 'cc-by-nc-sa',
         },
         'add_ie': ['Youtube'],
-    }, {
-        'url': 'https://www.khanacademy.org/math/applied-math/cryptography',
+    }
+
+    def _parse_component_props(self, component_props):
+        video = component_props['tutorialPageData']['contentModel']
+        info = self._parse_video(video)
+        author_names = video.get('authorNames')
+        info.update({
+            'uploader': ', '.join(author_names) if author_names else None,
+            'timestamp': parse_iso8601(video.get('dateAdded')),
+            'license': video.get('kaUserLicense'),
+        })
+        return info
+
+
+class KhanAcademyUnitIE(KhanAcademyBaseIE):
+    IE_NAME = 'khanacademy:unit'
+    _VALID_URL = (KhanAcademyBaseIE._VALID_URL_TEMPL % ('2', '')) + '/?(?:[?#&]|$)'
+    _TEST = {
+        'url': 'https://www.khanacademy.org/computing/computer-science/cryptography',
         'info_dict': {
             'id': 'cryptography',
-            'title': 'Journey into cryptography',
+            'title': 'Cryptography',
             'description': 'How have humans protected their secret messages through history? What has changed today?',
         },
-        'playlist_mincount': 3,
-    }]
+        'playlist_mincount': 31,
+    }
 
-    def _real_extract(self, url):
-        m = re.match(self._VALID_URL, url)
-        video_id = m.group('id')
+    def _parse_component_props(self, component_props):
+        curation = component_props['curation']
 
-        if m.group('key') == 'video':
-            data = self._download_json(
-                'http://api.khanacademy.org/api/v1/videos/' + video_id,
-                video_id, 'Downloading video info')
-
-            upload_date = unified_strdate(data['date_added'])
-            uploader = ', '.join(data['author_names'])
-            return {
-                '_type': 'url_transparent',
-                'url': data['url'],
-                'id': video_id,
-                'title': data['title'],
-                'thumbnail': data['image_url'],
-                'duration': data['duration'],
-                'description': data['description'],
-                'uploader': uploader,
-                'upload_date': upload_date,
+        entries = []
+        tutorials = try_get(curation, lambda x: x['tabs'][0]['modules'][0]['tutorials'], list) or []
+        for tutorial_number, tutorial in enumerate(tutorials, 1):
+            chapter_info = {
+                'chapter': tutorial.get('title'),
+                'chapter_number': tutorial_number,
+                'chapter_id': tutorial.get('id'),
             }
-        else:
-            # topic
-            data = self._download_json(
-                'http://api.khanacademy.org/api/v1/topic/' + video_id,
-                video_id, 'Downloading topic info')
+            for content_item in (tutorial.get('contentItems') or []):
+                if content_item.get('kind') == 'Video':
+                    info = self._parse_video(content_item)
+                    info.update(chapter_info)
+                    entries.append(info)
 
-            entries = [
-                {
-                    '_type': 'url',
-                    'url': c['url'],
-                    'id': c['id'],
-                    'title': c['title'],
-                }
-                for c in data['children'] if c['kind'] in ('Video', 'Topic')]
-
-            return {
-                '_type': 'playlist',
-                'id': video_id,
-                'title': data['title'],
-                'description': data['description'],
-                'entries': entries,
-            }
+        return self.playlist_result(
+            entries, curation.get('unit'), curation.get('title'),
+            curation.get('description'))

youtube_dlc/extractor/lbry.py

@@ -5,7 +5,10 @@ import functools
 import json
 
 from .common import InfoExtractor
-from ..compat import compat_str
+from ..compat import (
+    compat_str,
+    compat_urllib_parse_unquote,
+)
 from ..utils import (
     determine_ext,
     ExtractorError,
@@ -131,6 +134,9 @@ class LBRYIE(LBRYBaseIE):
     }, {
         'url': 'https://lbry.tv/$/download/Episode-1/e7d93d772bd87e2b62d5ab993c1c3ced86ebb396',
         'only_matching': True,
+    }, {
+        'url': 'https://lbry.tv/@lacajadepandora:a/TRUMP-EST%C3%81-BIEN-PUESTO-con-Pilar-Baselga,-Carlos-Senra,-Luis-Palacios-(720p_30fps_H264-192kbit_AAC):1',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -139,6 +145,7 @@ class LBRYIE(LBRYBaseIE):
             display_id = display_id.split('/', 2)[-1].replace('/', ':')
         else:
             display_id = display_id.replace(':', '#')
+        display_id = compat_urllib_parse_unquote(display_id)
         uri = 'lbry://' + display_id
         result = self._resolve_url(uri, display_id, 'stream')
         result_value = result['value']

youtube_dlc/extractor/mildom.py

@@ -0,0 +1,284 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from datetime import datetime
+import itertools
+import json
+import base64
+
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError, std_headers,
+    update_url_query,
+    random_uuidv4,
+    try_get,
+)
+from ..compat import (
+    compat_urlparse,
+    compat_urllib_parse_urlencode,
+    compat_str,
+)
+
+
+class MildomBaseIE(InfoExtractor):
+    _GUEST_ID = None
+    _DISPATCHER_CONFIG = None
+
+    def _call_api(self, url, video_id, query={}, note='Downloading JSON metadata', init=False):
+        url = update_url_query(url, self._common_queries(query, init=init))
+        return self._download_json(url, video_id, note=note)['body']
+
+    def _common_queries(self, query={}, init=False):
+        dc = self._fetch_dispatcher_config()
+        r = {
+            'timestamp': self.iso_timestamp(),
+            '__guest_id': '' if init else self.guest_id(),
+            '__location': dc['location'],
+            '__country': dc['country'],
+            '__cluster': dc['cluster'],
+            '__platform': 'web',
+            '__la': self.lang_code(),
+            '__pcv': 'v2.9.44',
+            'sfr': 'pc',
+            'accessToken': '',
+        }
+        r.update(query)
+        return r
+
+    def _fetch_dispatcher_config(self):
+        if not self._DISPATCHER_CONFIG:
+            try:
+                tmp = self._download_json(
+                    'https://disp.mildom.com/serverListV2', 'initialization',
+                    note='Downloading dispatcher_config', data=json.dumps({
+                        'protover': 0,
+                        'data': base64.b64encode(json.dumps({
+                            'fr': 'web',
+                            'sfr': 'pc',
+                            'devi': 'Windows',
+                            'la': 'ja',
+                            'gid': None,
+                            'loc': '',
+                            'clu': '',
+                            'wh': '1919*810',
+                            'rtm': self.iso_timestamp(),
+                            'ua': std_headers['User-Agent'],
+                        }).encode('utf8')).decode('utf8').replace('\n', ''),
+                    }).encode('utf8'))
+                self._DISPATCHER_CONFIG = self._parse_json(base64.b64decode(tmp['data']), 'initialization')
+            except ExtractorError:
+                self._DISPATCHER_CONFIG = self._download_json(
+                    'https://bookish-octo-barnacle.vercel.app/api/dispatcher_config', 'initialization',
+                    note='Downloading dispatcher_config fallback')
+        return self._DISPATCHER_CONFIG
+
+    @staticmethod
+    def iso_timestamp():
+        'new Date().toISOString()'
+        return datetime.utcnow().isoformat()[0:-3] + 'Z'
+
+    def guest_id(self):
+        'getGuestId'
+        if self._GUEST_ID:
+            return self._GUEST_ID
+        self._GUEST_ID = try_get(
+            self, (
+                lambda x: x._call_api(
+                    'https://cloudac.mildom.com/nonolive/gappserv/guest/h5init', 'initialization',
+                    note='Downloading guest token', init=True)['guest_id'] or None,
+                lambda x: x._get_cookies('https://www.mildom.com').get('gid').value,
+                lambda x: x._get_cookies('https://m.mildom.com').get('gid').value,
+            ), compat_str) or ''
+        return self._GUEST_ID
+
+    def lang_code(self):
+        'getCurrentLangCode'
+        return 'ja'
+
+
+class MildomIE(MildomBaseIE):
+    IE_NAME = 'mildom'
+    IE_DESC = 'Record ongoing live by specific user in Mildom'
+    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/(?P<id>\d+)'
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        url = 'https://www.mildom.com/%s' % video_id
+
+        webpage = self._download_webpage(url, video_id)
+
+        enterstudio = self._call_api(
+            'https://cloudac.mildom.com/nonolive/gappserv/live/enterstudio', video_id,
+            note='Downloading live metadata', query={'user_id': video_id})
+
+        title = try_get(
+            enterstudio, (
+                lambda x: self._html_search_meta('twitter:description', webpage),
+                lambda x: x['anchor_intro'],
+            ), compat_str)
+        description = try_get(
+            enterstudio, (
+                lambda x: x['intro'],
+                lambda x: x['live_intro'],
+            ), compat_str)
+        uploader = try_get(
+            enterstudio, (
+                lambda x: self._html_search_meta('twitter:title', webpage),
+                lambda x: x['loginname'],
+            ), compat_str)
+
+        servers = self._call_api(
+            'https://cloudac.mildom.com/nonolive/gappserv/live/liveserver', video_id,
+            note='Downloading live server list', query={
+                'user_id': video_id,
+                'live_server_type': 'hls',
+            })
+
+        stream_query = self._common_queries({
+            'streamReqId': random_uuidv4(),
+            'is_lhls': '0',
+        })
+        m3u8_url = update_url_query(servers['stream_server'] + '/%s_master.m3u8' % video_id, stream_query)
+        formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4', headers={
+            'Referer': 'https://www.mildom.com/',
+            'Origin': 'https://www.mildom.com',
+        }, note='Downloading m3u8 information')
+        del stream_query['streamReqId'], stream_query['timestamp']
+        for fmt in formats:
+            # Uses https://github.com/nao20010128nao/bookish-octo-barnacle by @nao20010128nao as a proxy
+            parsed = compat_urlparse.urlparse(fmt['url'])
+            parsed = parsed._replace(
+                netloc='bookish-octo-barnacle.vercel.app',
+                query=compat_urllib_parse_urlencode(stream_query, True),
+                path='/api' + parsed.path)
+            fmt['url'] = compat_urlparse.urlunparse(parsed)
+
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'uploader': uploader,
+            'uploader_id': video_id,
+            'formats': formats,
+            'is_live': True,
+        }
+
+
+class MildomVodIE(MildomBaseIE):
+    IE_NAME = 'mildom:vod'
+    IE_DESC = 'Download a VOD in Mildom'
+    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/playback/(?P<user_id>\d+)/(?P<id>(?P=user_id)-[a-zA-Z0-9]+)'
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        m = self._VALID_URL_RE.match(url)
+        user_id = m.group('user_id')
+        url = 'https://www.mildom.com/playback/%s/%s' % (user_id, video_id)
+
+        webpage = self._download_webpage(url, video_id)
+
+        autoplay = self._call_api(
+            'https://cloudac.mildom.com/nonolive/videocontent/playback/getPlaybackDetail', video_id,
+            note='Downloading playback metadata', query={
+                'v_id': video_id,
+            })['playback']
+
+        title = try_get(
+            autoplay, (
+                lambda x: self._html_search_meta('og:description', webpage),
+                lambda x: x['title'],
+            ), compat_str)
+        description = try_get(
+            autoplay, (
+                lambda x: x['video_intro'],
+            ), compat_str)
+        uploader = try_get(
+            autoplay, (
+                lambda x: x['author_info']['login_name'],
+            ), compat_str)
+
+        audio_formats = [{
+            'url': autoplay['audio_url'],
+            'format_id': 'audio',
+            'protocol': 'm3u8_native',
+            'vcodec': 'none',
+            'acodec': 'aac',
+        }]
+        video_formats = []
+        for fmt in autoplay['video_link']:
+            video_formats.append({
+                'format_id': 'video-%s' % fmt['name'],
+                'url': fmt['url'],
+                'protocol': 'm3u8_native',
+                'width': fmt['level'] * autoplay['video_width'] // autoplay['video_height'],
+                'height': fmt['level'],
+                'vcodec': 'h264',
+                'acodec': 'aac',
+            })
+
+        stream_query = self._common_queries({
+            'is_lhls': '0',
+        })
+        del stream_query['timestamp']
+        formats = audio_formats + video_formats
+        for fmt in formats:
+            fmt['ext'] = 'mp4'
+            parsed = compat_urlparse.urlparse(fmt['url'])
+            stream_query['path'] = parsed.path[5:]
+            parsed = parsed._replace(
+                netloc='bookish-octo-barnacle.vercel.app',
+                query=compat_urllib_parse_urlencode(stream_query, True),
+                path='/api/vod2/proxy')
+            fmt['url'] = compat_urlparse.urlunparse(parsed)
+
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'uploader': uploader,
+            'uploader_id': user_id,
+            'formats': formats,
+        }
+
+
+class MildomUserVodIE(MildomBaseIE):
+    IE_NAME = 'mildom:user:vod'
+    IE_DESC = 'Download all VODs from specific user in Mildom'
+    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/profile/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://www.mildom.com/profile/10093333',
+        'info_dict': {
+            'id': '10093333',
+            'title': 'Uploads from ねこばたけ',
+        },
+        'playlist_mincount': 351,
+    }]
+
+    def _real_extract(self, url):
+        user_id = self._match_id(url)
+
+        self._downloader.report_warning('To download ongoing live, please use "https://www.mildom.com/%s" instead. This will list up VODs belonging to user.' % user_id)
+
+        profile = self._call_api(
+            'https://cloudac.mildom.com/nonolive/gappserv/user/profileV2', user_id,
+            query={'user_id': user_id}, note='Downloading user profile')['user_info']
+
+        results = []
+        for page in itertools.count(1):
+            reply = self._call_api(
+                'https://cloudac.mildom.com/nonolive/videocontent/profile/playbackList',
+                user_id, note='Downloading page %d' % page, query={
+                    'user_id': user_id,
+                    'page': page,
+                    'limit': '30',
+                })
+            if not reply:
+                break
+            results.extend('https://www.mildom.com/playback/%s/%s' % (user_id, x['v_id']) for x in reply)
+        return self.playlist_result([
+            self.url_result(u, ie=MildomVodIE.ie_key()) for u in results
+        ], user_id, 'Uploads from %s' % profile['loginname'])

youtube_dlc/extractor/minds.py

@@ -0,0 +1,196 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..compat import compat_str
+from ..utils import (
+    clean_html,
+    int_or_none,
+    str_or_none,
+    strip_or_none,
+)
+
+
+class MindsBaseIE(InfoExtractor):
+    _VALID_URL_BASE = r'https?://(?:www\.)?minds\.com/'
+
+    def _call_api(self, path, video_id, resource, query=None):
+        api_url = 'https://www.minds.com/api/' + path
+        token = self._get_cookies(api_url).get('XSRF-TOKEN')
+        return self._download_json(
+            api_url, video_id, 'Downloading %s JSON metadata' % resource, headers={
+                'Referer': 'https://www.minds.com/',
+                'X-XSRF-TOKEN': token.value if token else '',
+            }, query=query)
+
+
+class MindsIE(MindsBaseIE):
+    IE_NAME = 'minds'
+    _VALID_URL = MindsBaseIE._VALID_URL_BASE + r'(?:media|newsfeed|archive/view)/(?P<id>[0-9]+)'
+    _TESTS = [{
+        'url': 'https://www.minds.com/media/100000000000086822',
+        'md5': '215a658184a419764852239d4970b045',
+        'info_dict': {
+            'id': '100000000000086822',
+            'ext': 'mp4',
+            'title': 'Minds intro sequence',
+            'thumbnail': r're:https?://.+\.png',
+            'uploader_id': 'ottman',
+            'upload_date': '20130524',
+            'timestamp': 1369404826,
+            'uploader': 'Bill Ottman',
+            'view_count': int,
+            'like_count': int,
+            'dislike_count': int,
+            'tags': ['animation'],
+            'comment_count': int,
+            'license': 'attribution-cc',
+        },
+    }, {
+        # entity.type == 'activity' and empty title
+        'url': 'https://www.minds.com/newsfeed/798025111988506624',
+        'md5': 'b2733a74af78d7fd3f541c4cbbaa5950',
+        'info_dict': {
+            'id': '798022190320226304',
+            'ext': 'mp4',
+            'title': '798022190320226304',
+            'uploader': 'ColinFlaherty',
+            'upload_date': '20180111',
+            'timestamp': 1515639316,
+            'uploader_id': 'ColinFlaherty',
+        },
+    }, {
+        'url': 'https://www.minds.com/archive/view/715172106794442752',
+        'only_matching': True,
+    }, {
+        # youtube perma_url
+        'url': 'https://www.minds.com/newsfeed/1197131838022602752',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        entity_id = self._match_id(url)
+        entity = self._call_api(
+            'v1/entities/entity/' + entity_id, entity_id, 'entity')['entity']
+        if entity.get('type') == 'activity':
+            if entity.get('custom_type') == 'video':
+                video_id = entity['entity_guid']
+            else:
+                return self.url_result(entity['perma_url'])
+        else:
+            assert(entity['subtype'] == 'video')
+            video_id = entity_id
+        # 1080p and webm formats available only on the sources array
+        video = self._call_api(
+            'v2/media/video/' + video_id, video_id, 'video')
+
+        formats = []
+        for source in (video.get('sources') or []):
+            src = source.get('src')
+            if not src:
+                continue
+            formats.append({
+                'format_id': source.get('label'),
+                'height': int_or_none(source.get('size')),
+                'url': src,
+            })
+        self._sort_formats(formats)
+
+        entity = video.get('entity') or entity
+        owner = entity.get('ownerObj') or {}
+        uploader_id = owner.get('username')
+
+        tags = entity.get('tags')
+        if tags and isinstance(tags, compat_str):
+            tags = [tags]
+
+        thumbnail = None
+        poster = video.get('poster') or entity.get('thumbnail_src')
+        if poster:
+            urlh = self._request_webpage(poster, video_id, fatal=False)
+            if urlh:
+                thumbnail = urlh.geturl()
+
+        return {
+            'id': video_id,
+            'title': entity.get('title') or video_id,
+            'formats': formats,
+            'description': clean_html(entity.get('description')) or None,
+            'license': str_or_none(entity.get('license')),
+            'timestamp': int_or_none(entity.get('time_created')),
+            'uploader': strip_or_none(owner.get('name')),
+            'uploader_id': uploader_id,
+            'uploader_url': 'https://www.minds.com/' + uploader_id if uploader_id else None,
+            'view_count': int_or_none(entity.get('play:count')),
+            'like_count': int_or_none(entity.get('thumbs:up:count')),
+            'dislike_count': int_or_none(entity.get('thumbs:down:count')),
+            'tags': tags,
+            'comment_count': int_or_none(entity.get('comments:count')),
+            'thumbnail': thumbnail,
+        }
+
+
+class MindsFeedBaseIE(MindsBaseIE):
+    _PAGE_SIZE = 150
+
+    def _entries(self, feed_id):
+        query = {'limit': self._PAGE_SIZE, 'sync': 1}
+        i = 1
+        while True:
+            data = self._call_api(
+                'v2/feeds/container/%s/videos' % feed_id,
+                feed_id, 'page %s' % i, query)
+            entities = data.get('entities') or []
+            for entity in entities:
+                guid = entity.get('guid')
+                if not guid:
+                    continue
+                yield self.url_result(
+                    'https://www.minds.com/newsfeed/' + guid,
+                    MindsIE.ie_key(), guid)
+            query['from_timestamp'] = data['load-next']
+            if not (query['from_timestamp'] and len(entities) == self._PAGE_SIZE):
+                break
+            i += 1
+
+    def _real_extract(self, url):
+        feed_id = self._match_id(url)
+        feed = self._call_api(
+            'v1/%s/%s' % (self._FEED_PATH, feed_id),
+            feed_id, self._FEED_TYPE)[self._FEED_TYPE]
+
+        return self.playlist_result(
+            self._entries(feed['guid']), feed_id,
+            strip_or_none(feed.get('name')),
+            feed.get('briefdescription'))
+
+
+class MindsChannelIE(MindsFeedBaseIE):
+    _FEED_TYPE = 'channel'
+    IE_NAME = 'minds:' + _FEED_TYPE
+    _VALID_URL = MindsBaseIE._VALID_URL_BASE + r'(?!(?:newsfeed|media|api|archive|groups)/)(?P<id>[^/?&#]+)'
+    _FEED_PATH = 'channel'
+    _TEST = {
+        'url': 'https://www.minds.com/ottman',
+        'info_dict': {
+            'id': 'ottman',
+            'title': 'Bill Ottman',
+            'description': 'Co-creator & CEO @minds',
+        },
+        'playlist_mincount': 54,
+    }
+
+
+class MindsGroupIE(MindsFeedBaseIE):
+    _FEED_TYPE = 'group'
+    IE_NAME = 'minds:' + _FEED_TYPE
+    _VALID_URL = MindsBaseIE._VALID_URL_BASE + r'groups/profile/(?P<id>[0-9]+)'
+    _FEED_PATH = 'groups/group'
+    _TEST = {
+        'url': 'https://www.minds.com/groups/profile/785582576369672204/feed/videos',
+        'info_dict': {
+            'id': '785582576369672204',
+            'title': 'Cooking Videos',
+        },
+        'playlist_mincount': 1,
+    }

youtube_dlc/extractor/mixcloud.py

@@ -251,8 +251,11 @@ class MixcloudPlaylistBaseIE(MixcloudBaseIE):
                 cloudcast_url = cloudcast.get('url')
                 if not cloudcast_url:
                     continue
+                slug = try_get(cloudcast, lambda x: x['slug'], compat_str)
+                owner_username = try_get(cloudcast, lambda x: x['owner']['username'], compat_str)
+                video_id = '%s_%s' % (owner_username, slug) if slug and owner_username else None
                 entries.append(self.url_result(
-                    cloudcast_url, MixcloudIE.ie_key(), cloudcast.get('slug')))
+                    cloudcast_url, MixcloudIE.ie_key(), video_id))
 
             page_info = items['pageInfo']
             has_next_page = page_info['hasNextPage']
@@ -321,7 +324,8 @@ class MixcloudUserIE(MixcloudPlaylistBaseIE):
     _DESCRIPTION_KEY = 'biog'
     _ROOT_TYPE = 'user'
     _NODE_TEMPLATE = '''slug
-          url'''
+          url
+          owner { username }'''
 
     def _get_playlist_title(self, title, slug):
         return '%s (%s)' % (title, slug)
@@ -345,6 +349,7 @@ class MixcloudPlaylistIE(MixcloudPlaylistBaseIE):
     _NODE_TEMPLATE = '''cloudcast {
             slug
             url
+            owner { username }
           }'''
 
     def _get_cloudcast(self, node):

youtube_dlc/extractor/mtv.py

@@ -255,6 +255,10 @@ class MTVServicesInfoExtractor(InfoExtractor):
 
         return try_get(feed, lambda x: x['result']['data']['id'], compat_str)
 
+    @staticmethod
+    def _extract_child_with_type(parent, t):
+        return next(c for c in parent['children'] if c.get('type') == t)
+
     def _extract_new_triforce_mgid(self, webpage, url='', video_id=None):
         if url == '':
             return
@@ -332,6 +336,13 @@ class MTVServicesInfoExtractor(InfoExtractor):
         if not mgid:
             mgid = self._extract_triforce_mgid(webpage, data_zone)
 
+        if not mgid:
+            data = self._parse_json(self._search_regex(
+                r'__DATA__\s*=\s*({.+?});', webpage, 'data'), None)
+            main_container = self._extract_child_with_type(data, 'MainContainer')
+            video_player = self._extract_child_with_type(main_container, 'VideoPlayer')
+            mgid = video_player['props']['media']['video']['config']['uri']
+
         return mgid
 
     def _real_extract(self, url):
@@ -403,18 +414,6 @@ class MTVIE(MTVServicesInfoExtractor):
         'only_matching': True,
     }]
 
-    @staticmethod
-    def extract_child_with_type(parent, t):
-        children = parent['children']
-        return next(c for c in children if c.get('type') == t)
-
-    def _extract_mgid(self, webpage):
-        data = self._parse_json(self._search_regex(
-            r'__DATA__\s*=\s*({.+?});', webpage, 'data'), None)
-        main_container = self.extract_child_with_type(data, 'MainContainer')
-        video_player = self.extract_child_with_type(main_container, 'VideoPlayer')
-        return video_player['props']['media']['video']['config']['uri']
-
 
 class MTVJapanIE(MTVServicesInfoExtractor):
     IE_NAME = 'mtvjapan'

youtube_dlc/extractor/ninegag.py

@@ -1,104 +1,125 @@
 from __future__ import unicode_literals
 
-import re
-
 from .common import InfoExtractor
-from ..utils import str_to_int
+from ..utils import (
+    determine_ext,
+    ExtractorError,
+    int_or_none,
+    try_get,
+    url_or_none,
+)
 
 
 class NineGagIE(InfoExtractor):
     IE_NAME = '9gag'
-    _VALID_URL = r'https?://(?:www\.)?9gag(?:\.com/tv|\.tv)/(?:p|embed)/(?P<id>[a-zA-Z0-9]+)(?:/(?P<display_id>[^?#/]+))?'
+    _VALID_URL = r'https?://(?:www\.)?9gag\.com/gag/(?P<id>[^/?&#]+)'
 
-    _TESTS = [{
-        'url': 'http://9gag.com/tv/p/Kk2X5/people-are-awesome-2013-is-absolutely-awesome',
+    _TEST = {
+        'url': 'https://9gag.com/gag/ae5Ag7B',
         'info_dict': {
-            'id': 'kXzwOKyGlSA',
+            'id': 'ae5Ag7B',
             'ext': 'mp4',
-            'description': 'This 3-minute video will make you smile and then make you feel untalented and insignificant. Anyway, you should share this awesomeness. (Thanks, Dino!)',
-            'title': '\"People Are Awesome 2013\" Is Absolutely Awesome',
-            'uploader_id': 'UCdEH6EjDKwtTe-sO2f0_1XA',
-            'uploader': 'CompilationChannel',
-            'upload_date': '20131110',
-            'view_count': int,
-        },
-        'add_ie': ['Youtube'],
-    }, {
-        'url': 'http://9gag.com/tv/p/aKolP3',
-        'info_dict': {
-            'id': 'aKolP3',
-            'ext': 'mp4',
-            'title': 'This Guy Travelled 11 countries In 44 days Just To Make This Amazing Video',
-            'description': "I just saw more in 1 minute than I've seen in 1 year. This guy's video is epic!!",
-            'uploader_id': 'rickmereki',
-            'uploader': 'Rick Mereki',
-            'upload_date': '20110803',
-            'view_count': int,
-        },
-        'add_ie': ['Vimeo'],
-    }, {
-        'url': 'http://9gag.com/tv/p/KklwM',
-        'only_matching': True,
-    }, {
-        'url': 'http://9gag.tv/p/Kk2X5',
-        'only_matching': True,
-    }, {
-        'url': 'http://9gag.com/tv/embed/a5Dmvl',
-        'only_matching': True,
-    }]
-
-    _EXTERNAL_VIDEO_PROVIDER = {
-        '1': {
-            'url': '%s',
-            'ie_key': 'Youtube',
-        },
-        '2': {
-            'url': 'http://player.vimeo.com/video/%s',
-            'ie_key': 'Vimeo',
-        },
-        '3': {
-            'url': 'http://instagram.com/p/%s',
-            'ie_key': 'Instagram',
-        },
-        '4': {
-            'url': 'http://vine.co/v/%s',
-            'ie_key': 'Vine',
-        },
+            'title': 'Capybara Agility Training',
+            'upload_date': '20191108',
+            'timestamp': 1573237208,
+            'categories': ['Awesome'],
+            'tags': ['Weimaraner', 'American Pit Bull Terrier'],
+            'duration': 44,
+            'like_count': int,
+            'dislike_count': int,
+            'comment_count': int,
+        }
     }
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('id')
-        display_id = mobj.group('display_id') or video_id
+        post_id = self._match_id(url)
+        post = self._download_json(
+            'https://9gag.com/v1/post', post_id, query={
+                'id': post_id
+            })['data']['post']
 
-        webpage = self._download_webpage(url, display_id)
+        if post.get('type') != 'Animated':
+            raise ExtractorError(
+                'The given url does not contain a video',
+                expected=True)
 
-        post_view = self._parse_json(
-            self._search_regex(
-                r'var\s+postView\s*=\s*new\s+app\.PostView\({\s*post:\s*({.+?})\s*,\s*posts:\s*prefetchedCurrentPost',
-                webpage, 'post view'),
-            display_id)
+        title = post['title']
 
-        ie_key = None
-        source_url = post_view.get('sourceUrl')
-        if not source_url:
-            external_video_id = post_view['videoExternalId']
-            external_video_provider = post_view['videoExternalProvider']
-            source_url = self._EXTERNAL_VIDEO_PROVIDER[external_video_provider]['url'] % external_video_id
-            ie_key = self._EXTERNAL_VIDEO_PROVIDER[external_video_provider]['ie_key']
-        title = post_view['title']
-        description = post_view.get('description')
-        view_count = str_to_int(post_view.get('externalView'))
-        thumbnail = post_view.get('thumbnail_700w') or post_view.get('ogImageUrl') or post_view.get('thumbnail_300w')
+        duration = None
+        formats = []
+        thumbnails = []
+        for key, image in (post.get('images') or {}).items():
+            image_url = url_or_none(image.get('url'))
+            if not image_url:
+                continue
+            ext = determine_ext(image_url)
+            image_id = key.strip('image')
+            common = {
+                'url': image_url,
+                'width': int_or_none(image.get('width')),
+                'height': int_or_none(image.get('height')),
+            }
+            if ext in ('jpg', 'png'):
+                webp_url = image.get('webpUrl')
+                if webp_url:
+                    t = common.copy()
+                    t.update({
+                        'id': image_id + '-webp',
+                        'url': webp_url,
+                    })
+                    thumbnails.append(t)
+                common.update({
+                    'id': image_id,
+                    'ext': ext,
+                })
+                thumbnails.append(common)
+            elif ext in ('webm', 'mp4'):
+                if not duration:
+                    duration = int_or_none(image.get('duration'))
+                common['acodec'] = 'none' if image.get('hasAudio') == 0 else None
+                for vcodec in ('vp8', 'vp9', 'h265'):
+                    c_url = image.get(vcodec + 'Url')
+                    if not c_url:
+                        continue
+                    c_f = common.copy()
+                    c_f.update({
+                        'format_id': image_id + '-' + vcodec,
+                        'url': c_url,
+                        'vcodec': vcodec,
+                    })
+                    formats.append(c_f)
+                common.update({
+                    'ext': ext,
+                    'format_id': image_id,
+                })
+                formats.append(common)
+        self._sort_formats(formats)
+
+        section = try_get(post, lambda x: x['postSection']['name'])
+
+        tags = None
+        post_tags = post.get('tags')
+        if post_tags:
+            tags = []
+            for tag in post_tags:
+                tag_key = tag.get('key')
+                if not tag_key:
+                    continue
+                tags.append(tag_key)
+
+        get_count = lambda x: int_or_none(post.get(x + 'Count'))
 
         return {
-            '_type': 'url_transparent',
-            'url': source_url,
-            'ie_key': ie_key,
-            'id': video_id,
-            'display_id': display_id,
+            'id': post_id,
             'title': title,
-            'description': description,
-            'view_count': view_count,
-            'thumbnail': thumbnail,
+            'timestamp': int_or_none(post.get('creationTs')),
+            'duration': duration,
+            'formats': formats,
+            'thumbnails': thumbnails,
+            'like_count': get_count('upVote'),
+            'dislike_count': get_count('downVote'),
+            'comment_count': get_count('comments'),
+            'age_limit': 18 if post.get('nsfw') == 1 else None,
+            'categories': [section] if section else None,
+            'tags': tags,
         }

youtube_dlc/extractor/njpwworld.py

@@ -6,30 +6,40 @@ import re
 from .common import InfoExtractor
 from ..compat import compat_urlparse
 from ..utils import (
-    extract_attributes,
     get_element_by_class,
     urlencode_postdata,
 )
 
 
 class NJPWWorldIE(InfoExtractor):
-    _VALID_URL = r'https?://njpwworld\.com/p/(?P<id>[a-z0-9_]+)'
+    _VALID_URL = r'https?://(front\.)?njpwworld\.com/p/(?P<id>[a-z0-9_]+)'
     IE_DESC = '新日本プロレスワールド'
     _NETRC_MACHINE = 'njpwworld'
 
-    _TEST = {
+    _TESTS = [{
         'url': 'http://njpwworld.com/p/s_series_00155_1_9/',
         'info_dict': {
             'id': 's_series_00155_1_9',
             'ext': 'mp4',
-            'title': '第9試合　ランディ・サベージ　vs　リック・スタイナー',
+            'title': '闘強導夢2000 2000年1月4日 東京ドーム 第9試合 ランディ・サベージ VS リック・スタイナー',
             'tags': list,
         },
         'params': {
             'skip_download': True,  # AES-encrypted m3u8
         },
         'skip': 'Requires login',
-    }
+    }, {
+        'url': 'https://front.njpwworld.com/p/s_series_00563_16_bs',
+        'info_dict': {
+            'id': 's_series_00563_16_bs',
+            'ext': 'mp4',
+            'title': 'WORLD TAG LEAGUE 2020 & BEST OF THE SUPER Jr.27 2020年12月6日 福岡・福岡国際センター バックステージコメント（字幕あり）',
+            'tags': ["福岡・福岡国際センター", "バックステージコメント", "2020", "20年代"],
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }]
 
     _LOGIN_URL = 'https://front.njpwworld.com/auth/login'
 
@@ -64,35 +74,27 @@ class NJPWWorldIE(InfoExtractor):
         webpage = self._download_webpage(url, video_id)
 
         formats = []
-        for mobj in re.finditer(r'<a[^>]+\bhref=(["\'])/player.+?[^>]*>', webpage):
-            player = extract_attributes(mobj.group(0))
-            player_path = player.get('href')
-            if not player_path:
-                continue
-            kind = self._search_regex(
-                r'(low|high)$', player.get('class') or '', 'kind',
-                default='low')
+        for kind, vid in re.findall(r'if\s+\(\s*imageQualityType\s*==\s*\'([^\']+)\'\s*\)\s*{\s*video_id\s*=\s*"(\d+)"', webpage):
+            player_path = '/intent?id=%s&type=url' % vid
             player_url = compat_urlparse.urljoin(url, player_path)
-            player_page = self._download_webpage(
-                player_url, video_id, note='Downloading player page')
-            entries = self._parse_html5_media_entries(
-                player_url, player_page, video_id, m3u8_id='hls-%s' % kind,
-                m3u8_entry_protocol='m3u8_native')
-            kind_formats = entries[0]['formats']
-            for f in kind_formats:
-                f['quality'] = 2 if kind == 'high' else 1
-            formats.extend(kind_formats)
+            formats.append({
+                'url': player_url,
+                'format_id': kind,
+                'ext': 'mp4',
+                'protocol': 'm3u8',
+                'quality': 2 if kind == 'high' else 1,
+            })
 
         self._sort_formats(formats)
 
-        post_content = get_element_by_class('post-content', webpage)
+        tag_block = get_element_by_class('tag-block', webpage)
         tags = re.findall(
-            r'<li[^>]+class="tag-[^"]+"><a[^>]*>([^<]+)</a></li>', post_content
-        ) if post_content else None
+            r'<a[^>]+class="tag-[^"]+"[^>]*>([^<]+)</a>', tag_block
+        ) if tag_block else None
 
         return {
             'id': video_id,
-            'title': self._og_search_title(webpage),
+            'title': get_element_by_class('article-title', webpage) or self._og_search_title(webpage),
             'formats': formats,
             'tags': tags,
         }

youtube_dlc/extractor/peertube.py

@@ -450,6 +450,18 @@ class PeerTubeIE(InfoExtractor):
             'tags': ['framasoft', 'peertube'],
             'categories': ['Science & Technology'],
         }
+    }, {
+        # Issue #26002
+        'url': 'peertube:spacepub.space:d8943b2d-8280-497b-85ec-bc282ec2afdc',
+        'info_dict': {
+            'id': 'd8943b2d-8280-497b-85ec-bc282ec2afdc',
+            'ext': 'mp4',
+            'title': 'Dot matrix printer shell demo',
+            'uploader_id': '3',
+            'timestamp': 1587401293,
+            'upload_date': '20200420',
+            'uploader': 'Drew DeVault',
+        }
     }, {
         'url': 'https://peertube.tamanoir.foucry.net/videos/watch/0b04f13d-1e18-4f1d-814e-4979aa7c9c44',
         'only_matching': True,
@@ -526,7 +538,15 @@ class PeerTubeIE(InfoExtractor):
         title = video['name']
 
         formats = []
-        for file_ in video['files']:
+        files = video.get('files') or []
+        for playlist in (video.get('streamingPlaylists') or []):
+            if not isinstance(playlist, dict):
+                continue
+            playlist_files = playlist.get('files')
+            if not (playlist_files and isinstance(playlist_files, list)):
+                continue
+            files.extend(playlist_files)
+        for file_ in files:
             if not isinstance(file_, dict):
                 continue
             file_url = url_or_none(file_.get('fileUrl'))

youtube_dlc/extractor/pokemon.py

@@ -75,7 +75,7 @@ class PokemonIE(InfoExtractor):
 
 
 class PokemonWatchIE(InfoExtractor):
-    _VALID_URL = r'https?://watch\.pokemon\.com/[a-z]{2}-[a-z]{2}/player\.html\?id=(?P<id>[a-z0-9]{32})'
+    _VALID_URL = r'https?://watch\.pokemon\.com/[a-z]{2}-[a-z]{2}/(?:#/)?player(?:\.html)?\?id=(?P<id>[a-z0-9]{32})'
     _API_URL = 'https://www.pokemon.com/api/pokemontv/v2/channels/{0:}'
     _TESTS = [{
         'url': 'https://watch.pokemon.com/en-us/player.html?id=8309a40969894a8e8d5bc1311e9c5667',
@@ -86,6 +86,9 @@ class PokemonWatchIE(InfoExtractor):
             'title': 'Lillier and the Staff!',
             'description': 'md5:338841b8c21b283d24bdc9b568849f04',
         }
+    }, {
+        'url': 'https://watch.pokemon.com/en-us/#/player?id=3fe7752ba09141f0b0f7756d1981c6b2',
+        'only_matching': True
     }, {
         'url': 'https://watch.pokemon.com/de-de/player.html?id=b3c402e111a4459eb47e12160ab0ba07',
         'only_matching': True

youtube_dlc/extractor/roosterteeth.py

@@ -30,6 +30,19 @@ class RoosterTeethIE(InfoExtractor):
             'series': 'Million Dollars, But...',
             'episode': 'Million Dollars, But... The Game Announcement',
         },
+    }, {
+        'url': 'https://roosterteeth.com/watch/rwby-bonus-25',
+        'md5': 'fe8d9d976b272c18a24fe7f1f5830084',
+        'info_dict': {
+            'id': '31',
+            'display_id': 'rwby-bonus-25',
+            'title': 'Volume 2, World of Remnant 3',
+            'description': 'md5:8d58d3270292ea11da00ea712bbfb009',
+            'episode': 'Volume 2, World of Remnant 3',
+            'channel_id': 'fab60c1c-29cb-43bc-9383-5c3538d9e246',
+            'thumbnail': r're:^https?://.*\.(png|jpe?g)$',
+            'ext': 'mp4',
+        },
     }, {
         'url': 'http://achievementhunter.roosterteeth.com/episode/off-topic-the-achievement-hunter-podcast-2016-i-didn-t-think-it-would-pass-31',
         'only_matching': True,
@@ -50,7 +63,7 @@ class RoosterTeethIE(InfoExtractor):
         'url': 'https://roosterteeth.com/watch/million-dollars-but-season-2-million-dollars-but-the-game-announcement',
         'only_matching': True,
     }]
-    _EPISODE_BASE_URL = 'https://svod-be.roosterteeth.com/api/v1/episodes/'
+    _EPISODE_BASE_URL = 'https://svod-be.roosterteeth.com/api/v1/watch/'
 
     def _login(self):
         username, password = self._get_login_info()
@@ -111,7 +124,7 @@ class RoosterTeethIE(InfoExtractor):
 
         thumbnails = []
         for image in episode.get('included', {}).get('images', []):
-            if image.get('type') == 'episode_image':
+            if image.get('type') in ('episode_image', 'bonus_feature_image'):
                 img_attributes = image.get('attributes') or {}
                 for k in ('thumb', 'small', 'medium', 'large'):
                     img_url = img_attributes.get(k)

youtube_dlc/extractor/spike.py

@@ -20,19 +20,6 @@ class BellatorIE(MTVServicesInfoExtractor):
     _FEED_URL = 'http://www.bellator.com/feeds/mrss/'
     _GEO_COUNTRIES = ['US']
 
-    def _extract_mgid(self, webpage, url):
-        mgid = None
-
-        if not mgid:
-            mgid = self._extract_triforce_mgid(webpage)
-
-        if not mgid:
-            mgid = self._extract_new_triforce_mgid(webpage, url)
-
-        return mgid
-
-# TODO Remove - Reason: Outdated Site
-
 
 class ParamountNetworkIE(MTVServicesInfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?paramountnetwork\.com/[^/]+/[\da-z]{6}(?:[/?#&]|$)'
@@ -50,16 +37,12 @@ class ParamountNetworkIE(MTVServicesInfoExtractor):
         },
     }]
 
-    _FEED_URL = 'http://www.paramountnetwork.com/feeds/mrss/'
+    _FEED_URL = 'http://feeds.mtvnservices.com/od/feed/intl-mrss-player-feed'
     _GEO_COUNTRIES = ['US']
 
-    def _extract_mgid(self, webpage, url):
-        root_data = self._parse_json(self._search_regex(
-            r'window\.__DATA__\s*=\s*({.+})',
-            webpage, 'data'), None)
-
-        def find_sub_data(data, data_type):
-            return next(c for c in data['children'] if c.get('type') == data_type)
-
-        c = find_sub_data(find_sub_data(root_data, 'MainContainer'), 'VideoPlayer')
-        return c['props']['media']['video']['config']['uri']
+    def _get_feed_query(self, uri):
+        return {
+            'arcEp': 'paramountnetwork.com',
+            'imageEp': 'paramountnetwork.com',
+            'mgid': uri,
+        }

youtube_dlc/extractor/spotify.py

@@ -0,0 +1,156 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    clean_podcast_url,
+    float_or_none,
+    int_or_none,
+    strip_or_none,
+    try_get,
+    unified_strdate,
+)
+
+
+class SpotifyBaseIE(InfoExtractor):
+    _ACCESS_TOKEN = None
+    _OPERATION_HASHES = {
+        'Episode': '8276d4423d709ae9b68ec1b74cc047ba0f7479059a37820be730f125189ac2bf',
+        'MinimalShow': '13ee079672fad3f858ea45a55eb109553b4fb0969ed793185b2e34cbb6ee7cc0',
+        'ShowEpisodes': 'e0e5ce27bd7748d2c59b4d44ba245a8992a05be75d6fabc3b20753fc8857444d',
+    }
+    _VALID_URL_TEMPL = r'https?://open\.spotify\.com/%s/(?P<id>[^/?&#]+)'
+
+    def _real_initialize(self):
+        self._ACCESS_TOKEN = self._download_json(
+            'https://open.spotify.com/get_access_token', None)['accessToken']
+
+    def _call_api(self, operation, video_id, variables):
+        return self._download_json(
+            'https://api-partner.spotify.com/pathfinder/v1/query', video_id, query={
+                'operationName': 'query' + operation,
+                'variables': json.dumps(variables),
+                'extensions': json.dumps({
+                    'persistedQuery': {
+                        'sha256Hash': self._OPERATION_HASHES[operation],
+                    },
+                })
+            }, headers={'authorization': 'Bearer ' + self._ACCESS_TOKEN})['data']
+
+    def _extract_episode(self, episode, series):
+        episode_id = episode['id']
+        title = episode['name'].strip()
+
+        formats = []
+        audio_preview = episode.get('audioPreview') or {}
+        audio_preview_url = audio_preview.get('url')
+        if audio_preview_url:
+            f = {
+                'url': audio_preview_url.replace('://p.scdn.co/mp3-preview/', '://anon-podcast.scdn.co/'),
+                'vcodec': 'none',
+            }
+            audio_preview_format = audio_preview.get('format')
+            if audio_preview_format:
+                f['format_id'] = audio_preview_format
+                mobj = re.match(r'([0-9A-Z]{3})_(?:[A-Z]+_)?(\d+)', audio_preview_format)
+                if mobj:
+                    f.update({
+                        'abr': int(mobj.group(2)),
+                        'ext': mobj.group(1).lower(),
+                    })
+            formats.append(f)
+
+        for item in (try_get(episode, lambda x: x['audio']['items']) or []):
+            item_url = item.get('url')
+            if not (item_url and item.get('externallyHosted')):
+                continue
+            formats.append({
+                'url': clean_podcast_url(item_url),
+                'vcodec': 'none',
+            })
+
+        thumbnails = []
+        for source in (try_get(episode, lambda x: x['coverArt']['sources']) or []):
+            source_url = source.get('url')
+            if not source_url:
+                continue
+            thumbnails.append({
+                'url': source_url,
+                'width': int_or_none(source.get('width')),
+                'height': int_or_none(source.get('height')),
+            })
+
+        return {
+            'id': episode_id,
+            'title': title,
+            'formats': formats,
+            'thumbnails': thumbnails,
+            'description': strip_or_none(episode.get('description')),
+            'duration': float_or_none(try_get(
+                episode, lambda x: x['duration']['totalMilliseconds']), 1000),
+            'release_date': unified_strdate(try_get(
+                episode, lambda x: x['releaseDate']['isoString'])),
+            'series': series,
+        }
+
+
+class SpotifyIE(SpotifyBaseIE):
+    IE_NAME = 'spotify'
+    _VALID_URL = SpotifyBaseIE._VALID_URL_TEMPL % 'episode'
+    _TEST = {
+        'url': 'https://open.spotify.com/episode/4Z7GAJ50bgctf6uclHlWKo',
+        'md5': '74010a1e3fa4d9e1ab3aa7ad14e42d3b',
+        'info_dict': {
+            'id': '4Z7GAJ50bgctf6uclHlWKo',
+            'ext': 'mp3',
+            'title': 'From the archive: Why time management is ruining our lives',
+            'description': 'md5:b120d9c4ff4135b42aa9b6d9cde86935',
+            'duration': 2083.605,
+            'release_date': '20201217',
+            'series': "The Guardian's Audio Long Reads",
+        }
+    }
+
+    def _real_extract(self, url):
+        episode_id = self._match_id(url)
+        episode = self._call_api('Episode', episode_id, {
+            'uri': 'spotify:episode:' + episode_id
+        })['episode']
+        return self._extract_episode(
+            episode, try_get(episode, lambda x: x['podcast']['name']))
+
+
+class SpotifyShowIE(SpotifyBaseIE):
+    IE_NAME = 'spotify:show'
+    _VALID_URL = SpotifyBaseIE._VALID_URL_TEMPL % 'show'
+    _TEST = {
+        'url': 'https://open.spotify.com/show/4PM9Ke6l66IRNpottHKV9M',
+        'info_dict': {
+            'id': '4PM9Ke6l66IRNpottHKV9M',
+            'title': 'The Story from the Guardian',
+            'description': 'The Story podcast is dedicated to our finest audio documentaries, investigations and long form stories',
+        },
+        'playlist_mincount': 36,
+    }
+
+    def _real_extract(self, url):
+        show_id = self._match_id(url)
+        podcast = self._call_api('ShowEpisodes', show_id, {
+            'limit': 1000000000,
+            'offset': 0,
+            'uri': 'spotify:show:' + show_id,
+        })['podcast']
+        podcast_name = podcast.get('name')
+
+        entries = []
+        for item in (try_get(podcast, lambda x: x['episodes']['items']) or []):
+            episode = item.get('episode')
+            if not episode:
+                continue
+            entries.append(self._extract_episode(episode, podcast_name))
+
+        return self.playlist_result(
+            entries, show_id, podcast_name, podcast.get('description'))

youtube_dlc/extractor/threeqsdn.py

@@ -3,10 +3,13 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
+from ..compat import compat_HTTPError
 from ..utils import (
     determine_ext,
-    js_to_json,
-    mimetype2ext,
+    ExtractorError,
+    float_or_none,
+    int_or_none,
+    parse_iso8601,
 )
 
 
@@ -15,29 +18,35 @@ class ThreeQSDNIE(InfoExtractor):
     IE_DESC = '3Q SDN'
     _VALID_URL = r'https?://playout\.3qsdn\.com/(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
     _TESTS = [{
-        # ondemand from http://www.philharmonie.tv/veranstaltung/26/
-        'url': 'http://playout.3qsdn.com/0280d6b9-1215-11e6-b427-0cc47a188158?protocol=http',
-        'md5': 'ab040e37bcfa2e0c079f92cb1dd7f6cd',
+        # https://player.3qsdn.com/demo.html
+        'url': 'https://playout.3qsdn.com/7201c779-6b3c-11e7-a40e-002590c750be',
+        'md5': '64a57396b16fa011b15e0ea60edce918',
         'info_dict': {
-            'id': '0280d6b9-1215-11e6-b427-0cc47a188158',
+            'id': '7201c779-6b3c-11e7-a40e-002590c750be',
             'ext': 'mp4',
-            'title': '0280d6b9-1215-11e6-b427-0cc47a188158',
+            'title': 'Video Ads',
             'is_live': False,
+            'description': 'Video Ads Demo',
+            'timestamp': 1500334803,
+            'upload_date': '20170717',
+            'duration': 888.032,
+            'subtitles': {
+                'eng': 'count:1',
+            },
         },
-        'expected_warnings': ['Failed to download MPD manifest', 'Failed to parse JSON'],
+        'expected_warnings': ['Unknown MIME type application/mp4 in DASH manifest'],
     }, {
         # live video stream
-        'url': 'https://playout.3qsdn.com/d755d94b-4ab9-11e3-9162-0025907ad44f?js=true',
+        'url': 'https://playout.3qsdn.com/66e68995-11ca-11e8-9273-002590c750be',
         'info_dict': {
-            'id': 'd755d94b-4ab9-11e3-9162-0025907ad44f',
+            'id': '66e68995-11ca-11e8-9273-002590c750be',
             'ext': 'mp4',
-            'title': 're:^d755d94b-4ab9-11e3-9162-0025907ad44f [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
+            'title': 're:^66e68995-11ca-11e8-9273-002590c750be [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
             'is_live': True,
         },
         'params': {
             'skip_download': True,  # m3u8 downloads
         },
-        'expected_warnings': ['Failed to download MPD manifest'],
     }, {
         # live audio stream
         'url': 'http://playout.3qsdn.com/9edf36e0-6bf2-11e2-a16a-9acf09e2db48',
@@ -58,6 +67,14 @@ class ThreeQSDNIE(InfoExtractor):
         # live video with rtmp link
         'url': 'https://playout.3qsdn.com/6092bb9e-8f72-11e4-a173-002590c750be',
         'only_matching': True,
+    }, {
+        # ondemand from http://www.philharmonie.tv/veranstaltung/26/
+        'url': 'http://playout.3qsdn.com/0280d6b9-1215-11e6-b427-0cc47a188158?protocol=http',
+        'only_matching': True,
+    }, {
+        # live video stream
+        'url': 'https://playout.3qsdn.com/d755d94b-4ab9-11e3-9162-0025907ad44f?js=true',
+        'only_matching': True,
     }]
 
     @staticmethod
@@ -70,73 +87,78 @@ class ThreeQSDNIE(InfoExtractor):
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        js = self._download_webpage(
-            'http://playout.3qsdn.com/%s' % video_id, video_id,
-            query={'js': 'true'})
+        try:
+            config = self._download_json(
+                url.replace('://playout.3qsdn.com/', '://playout.3qsdn.com/config/'), video_id)
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
+                self.raise_geo_restricted()
+            raise
 
-        if any(p in js for p in (
-                '>This content is not available in your country',
-                'playout.3qsdn.com/forbidden')):
-            self.raise_geo_restricted()
-
-        stream_content = self._search_regex(
-            r'streamContent\s*:\s*(["\'])(?P<content>.+?)\1', js,
-            'stream content', default='demand', group='content')
-
-        live = stream_content == 'live'
-
-        stream_type = self._search_regex(
-            r'streamType\s*:\s*(["\'])(?P<type>audio|video)\1', js,
-            'stream type', default='video', group='type')
+        live = config.get('streamContent') == 'live'
+        aspect = float_or_none(config.get('aspect'))
 
         formats = []
-        urls = set()
-
-        def extract_formats(item_url, item={}):
-            if not item_url or item_url in urls:
-                return
-            urls.add(item_url)
-            ext = mimetype2ext(item.get('type')) or determine_ext(item_url, default_ext=None)
-            if ext == 'mpd':
-                formats.extend(self._extract_mpd_formats(
-                    item_url, video_id, mpd_id='mpd', fatal=False))
-            elif ext == 'm3u8':
-                formats.extend(self._extract_m3u8_formats(
-                    item_url, video_id, 'mp4',
-                    entry_protocol='m3u8' if live else 'm3u8_native',
-                    m3u8_id='hls', fatal=False))
-            elif ext == 'f4m':
-                formats.extend(self._extract_f4m_formats(
-                    item_url, video_id, f4m_id='hds', fatal=False))
-            else:
-                if not self._is_valid_url(item_url, video_id):
-                    return
-                formats.append({
-                    'url': item_url,
-                    'format_id': item.get('quality'),
-                    'ext': 'mp4' if item_url.startswith('rtsp') else ext,
-                    'vcodec': 'none' if stream_type == 'audio' else None,
-                })
-
-        for item_js in re.findall(r'({[^{]*?\b(?:src|source)\s*:\s*["\'].+?})', js):
-            f = self._parse_json(
-                item_js, video_id, transform_source=js_to_json, fatal=False)
-            if not f:
+        for source_type, source in (config.get('sources') or {}).items():
+            if not source:
                 continue
-            extract_formats(f.get('src'), f)
+            if source_type == 'dash':
+                formats.extend(self._extract_mpd_formats(
+                    source, video_id, mpd_id='mpd', fatal=False))
+            elif source_type == 'hls':
+                formats.extend(self._extract_m3u8_formats(
+                    source, video_id, 'mp4', 'm3u8' if live else 'm3u8_native',
+                    m3u8_id='hls', fatal=False))
+            elif source_type == 'progressive':
+                for s in source:
+                    src = s.get('src')
+                    if not (src and self._is_valid_url(src, video_id)):
+                        continue
+                    width = None
+                    format_id = ['http']
+                    ext = determine_ext(src)
+                    if ext:
+                        format_id.append(ext)
+                    height = int_or_none(s.get('height'))
+                    if height:
+                        format_id.append('%dp' % height)
+                        if aspect:
+                            width = int(height * aspect)
+                    formats.append({
+                        'ext': ext,
+                        'format_id': '-'.join(format_id),
+                        'height': height,
+                        'source_preference': 0,
+                        'url': src,
+                        'vcodec': 'none' if height == 0 else None,
+                        'width': width,
+                    })
+        for f in formats:
+            if f.get('acodec') == 'none':
+                f['preference'] = -40
+            elif f.get('vcodec') == 'none':
+                f['preference'] = -50
+        self._sort_formats(formats, ('preference', 'width', 'height', 'source_preference', 'tbr', 'vbr', 'abr', 'ext', 'format_id'))
 
-        # More relaxed version to collect additional URLs and acting
-        # as a future-proof fallback
-        for _, src in re.findall(r'\b(?:src|source)\s*:\s*(["\'])((?:https?|rtsp)://.+?)\1', js):
-            extract_formats(src)
+        subtitles = {}
+        for subtitle in (config.get('subtitles') or []):
+            src = subtitle.get('src')
+            if not src:
+                continue
+            subtitles.setdefault(subtitle.get('label') or 'eng', []).append({
+                'url': src,
+            })
 
-        self._sort_formats(formats)
-
-        title = self._live_title(video_id) if live else video_id
+        title = config.get('title') or video_id
 
         return {
             'id': video_id,
-            'title': title,
+            'title': self._live_title(title) if live else title,
+            'thumbnail': config.get('poster') or None,
+            'description': config.get('description') or None,
+            'timestamp': parse_iso8601(config.get('upload_date')),
+            'duration': float_or_none(config.get('vlength')) or None,
             'is_live': live,
             'formats': formats,
+            'subtitles': subtitles,
         }

youtube_dlc/extractor/tiktok.py

@@ -17,8 +17,8 @@ class TikTokBaseIE(InfoExtractor):
         video_info = try_get(
             video_data, lambda x: x['itemInfo']['itemStruct'], dict)
         author_info = try_get(
-            video_data, lambda x: x['itemInfo']['itemStruct']['author'], dict)
-        share_info = try_get(video_data, lambda x: x['itemInfo']['shareMeta'], dict)
+            video_data, lambda x: x['itemInfo']['itemStruct']['author'], dict) or {}
+        share_info = try_get(video_data, lambda x: x['itemInfo']['shareMeta'], dict) or {}
 
         unique_id = str_or_none(author_info.get('uniqueId'))
         timestamp = try_get(video_info, lambda x: int(x['createTime']), int)

youtube_dlc/extractor/trovo.py

@@ -0,0 +1,193 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError,
+    int_or_none,
+    str_or_none,
+    try_get,
+)
+
+
+class TrovoBaseIE(InfoExtractor):
+    _VALID_URL_BASE = r'https?://(?:www\.)?trovo\.live/'
+
+    def _extract_streamer_info(self, data):
+        streamer_info = data.get('streamerInfo') or {}
+        username = streamer_info.get('userName')
+        return {
+            'uploader': streamer_info.get('nickName'),
+            'uploader_id': str_or_none(streamer_info.get('uid')),
+            'uploader_url': 'https://trovo.live/' + username if username else None,
+        }
+
+
+class TrovoIE(TrovoBaseIE):
+    _VALID_URL = TrovoBaseIE._VALID_URL_BASE + r'(?!(?:clip|video)/)(?P<id>[^/?&#]+)'
+
+    def _real_extract(self, url):
+        username = self._match_id(url)
+        live_info = self._download_json(
+            'https://gql.trovo.live/', username, query={
+                'query': '''{
+  getLiveInfo(params: {userName: "%s"}) {
+    isLive
+    programInfo	{
+      coverUrl
+      id
+      streamInfo {
+        desc
+        playUrl
+      }
+      title
+    }
+    streamerInfo {
+        nickName
+        uid
+        userName
+    }
+  }
+}''' % username,
+            })['data']['getLiveInfo']
+        if live_info.get('isLive') == 0:
+            raise ExtractorError('%s is offline' % username, expected=True)
+        program_info = live_info['programInfo']
+        program_id = program_info['id']
+        title = self._live_title(program_info['title'])
+
+        formats = []
+        for stream_info in (program_info.get('streamInfo') or []):
+            play_url = stream_info.get('playUrl')
+            if not play_url:
+                continue
+            format_id = stream_info.get('desc')
+            formats.append({
+                'format_id': format_id,
+                'height': int_or_none(format_id[:-1]) if format_id else None,
+                'url': play_url,
+            })
+        self._sort_formats(formats)
+
+        info = {
+            'id': program_id,
+            'title': title,
+            'formats': formats,
+            'thumbnail': program_info.get('coverUrl'),
+            'is_live': True,
+        }
+        info.update(self._extract_streamer_info(live_info))
+        return info
+
+
+class TrovoVodIE(TrovoBaseIE):
+    _VALID_URL = TrovoBaseIE._VALID_URL_BASE + r'(?:clip|video)/(?P<id>[^/?&#]+)'
+    _TESTS = [{
+        'url': 'https://trovo.live/video/ltv-100095501_100095501_1609596043',
+        'info_dict': {
+            'id': 'ltv-100095501_100095501_1609596043',
+            'ext': 'mp4',
+            'title': 'Spontaner 12 Stunden Stream! - Ok Boomer!',
+            'uploader': 'Exsl',
+            'timestamp': 1609640305,
+            'upload_date': '20210103',
+            'uploader_id': '100095501',
+            'duration': 43977,
+            'view_count': int,
+            'like_count': int,
+            'comment_count': int,
+            'comments': 'mincount:8',
+            'categories': ['Grand Theft Auto V'],
+        },
+    }, {
+        'url': 'https://trovo.live/clip/lc-5285890810184026005',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        vid = self._match_id(url)
+        resp = self._download_json(
+            'https://gql.trovo.live/', vid, data=json.dumps([{
+                'query': '''{
+  batchGetVodDetailInfo(params: {vids: ["%s"]}) {
+    VodDetailInfos
+  }
+}''' % vid,
+            }, {
+                'query': '''{
+  getCommentList(params: {appInfo: {postID: "%s"}, pageSize: 1000000000, preview: {}}) {
+    commentList {
+      author {
+        nickName
+        uid
+      }
+      commentID
+      content
+      createdAt
+      parentID
+    }
+  }
+}''' % vid,
+            }]).encode(), headers={
+                'Content-Type': 'application/json',
+            })
+        vod_detail_info = resp[0]['data']['batchGetVodDetailInfo']['VodDetailInfos'][vid]
+        vod_info = vod_detail_info['vodInfo']
+        title = vod_info['title']
+
+        language = vod_info.get('languageName')
+        formats = []
+        for play_info in (vod_info.get('playInfos') or []):
+            play_url = play_info.get('playUrl')
+            if not play_url:
+                continue
+            format_id = play_info.get('desc')
+            formats.append({
+                'ext': 'mp4',
+                'filesize': int_or_none(play_info.get('fileSize')),
+                'format_id': format_id,
+                'height': int_or_none(format_id[:-1]) if format_id else None,
+                'language': language,
+                'protocol': 'm3u8_native',
+                'tbr': int_or_none(play_info.get('bitrate')),
+                'url': play_url,
+            })
+        self._sort_formats(formats)
+
+        category = vod_info.get('categoryName')
+        get_count = lambda x: int_or_none(vod_info.get(x + 'Num'))
+
+        comment_list = try_get(resp, lambda x: x[1]['data']['getCommentList']['commentList'], list) or []
+        comments = []
+        for comment in comment_list:
+            content = comment.get('content')
+            if not content:
+                continue
+            author = comment.get('author') or {}
+            parent = comment.get('parentID')
+            comments.append({
+                'author': author.get('nickName'),
+                'author_id': str_or_none(author.get('uid')),
+                'id': str_or_none(comment.get('commentID')),
+                'text': content,
+                'timestamp': int_or_none(comment.get('createdAt')),
+                'parent': 'root' if parent == 0 else str_or_none(parent),
+            })
+
+        info = {
+            'id': vid,
+            'title': title,
+            'formats': formats,
+            'thumbnail': vod_info.get('coverUrl'),
+            'timestamp': int_or_none(vod_info.get('publishTs')),
+            'duration': int_or_none(vod_info.get('duration')),
+            'view_count': get_count('watch'),
+            'like_count': get_count('like'),
+            'comment_count': get_count('comment'),
+            'comments': comments,
+            'categories': [category] if category else None,
+        }
+        info.update(self._extract_streamer_info(vod_detail_info))
+        return info

youtube_dlc/extractor/trovolive.py

@@ -0,0 +1,111 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+
+from ..utils import (
+    js_to_json,
+    try_get,
+    int_or_none,
+    str_or_none,
+    url_or_none,
+)
+from ..compat import compat_str
+
+
+class TrovoLiveIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?trovo\.live/video/(?P<id>[\w-]+)'
+    _TEST = {
+        'url': 'https://trovo.live/video/ltv-100759829_100759829_1610625308',
+        'md5': 'ea7b58427910e9af66a462d895201a30',
+        'info_dict': {
+            'id': 'ltv-100759829_100759829_1610625308',
+            'ext': 'ts',
+            'title': 'GTA RP ASTERIX doa najjaca',
+            'uploader': 'Peroo42',
+            'duration': 5872,
+            'view_count': int,
+            'like_count': int,
+            'comment_count': int,
+            'categories': list,
+            'is_live': False,
+            'thumbnail': r're:^https?://.*\.jpg$',
+            'uploader_id': '100759829',
+        }
+    }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        nuxt = self._search_regex(r'\bwindow\.__NUXT__\s*=\s*(.+?);?\s*</script>', webpage, 'nuxt', default='')
+        mobj = re.search(r'\((?P<arg_names>[^(]+)\)\s*{\s*return\s+(?P<json>{.+})\s*\((?P<args>.+?)\)\s*\)$', nuxt)
+
+        vod_details = vod_info = {}
+        if mobj:
+            vod_details = self._parse_json(
+                js_to_json(
+                    self._search_regex(r'VodDetailInfos\s*:({.+?}),\s*_', webpage, 'VodDetailInfos'),
+                    dict(zip(
+                        (i.strip() for i in mobj.group('arg_names').split(',')),
+                        (i.strip() for i in mobj.group('args').split(','))))),
+                video_id)
+            vod_info = try_get(vod_details, lambda x: x['json'][video_id]['vodInfo'], dict) or {}
+
+        player_info = self._parse_json(
+            self._search_regex(
+                r'_playerInfo\s*=\s*({.+?})\s*</script>', webpage, 'player info'),
+            video_id)
+
+        title = (
+            vod_info.get('title')
+            or self._html_search_regex(r'<h3>(.+?)</h3>', webpage, 'title', fatal=False)
+            or self._og_search_title(webpage))
+        uploader = (
+            try_get(vod_details, lambda x: x['json'][video_id]['streamerInfo']['userName'], compat_str)
+            or self._search_regex(r'<div[^>]+userName\s=\s[\'"](.+?)[\'"]', webpage, 'uploader', fatal=False))
+
+        format_dicts = vod_info.get('playInfos') or player_info.get('urlArray') or []
+
+        def _extract_format_data(format_dict):
+            res = format_dict.get('desc')
+            enc = str_or_none(format_dict.get('encodeType'))
+            if enc:
+                notes = [enc.replace('VOD_ENCODE_TYPE_', '')]
+            level = str_or_none(format_dict.get('levelType'))
+            if level:
+                notes.append('level %s' % level)
+            height = int_or_none(res[:-1]) if res else None
+            bitrate = format_dict.get('bitrate')
+            fid = res or ('%sk' % str_or_none(bitrate) if bitrate else None) or ' '.join(notes)
+
+            return {
+                'url': format_dict['playUrl'],
+                'format_id': fid,
+                'format_note': ' '.join(notes),
+                'height': height,
+                'resolution': str_or_none(res),
+                'tbr': int_or_none(bitrate),
+                'filesize': int_or_none(format_dict.get('fileSize')),
+                'vcodec': 'avc3',
+                'acodec': 'aac',
+                'ext': 'ts'
+            }
+
+        formats = [_extract_format_data(f) for f in format_dicts]
+        self._sort_formats(formats)
+        return {
+            'id': video_id,
+            'title': title,
+            'uploader': uploader,
+            'duration': int_or_none(vod_info.get('duration')),
+            'formats': formats,
+            'view_count': int_or_none(vod_info.get('watchNum')),
+            'like_count': int_or_none(vod_info.get('likeNum')),
+            'comment_count': int_or_none(vod_info.get('commentNum')),
+            'categories': [str_or_none(vod_info.get('categoryName'))],
+            'is_live': try_get(player_info, lambda x: x['isLive'], bool),
+            'thumbnail': url_or_none(vod_info.get('coverUrl')),
+            'uploader_id': str_or_none(try_get(vod_details, lambda x: x['json'][video_id]['streamerInfo']['uid'])),
+        }

youtube_dlc/extractor/twitch.py

@@ -17,6 +17,7 @@ from ..compat import (
 )
 from ..utils import (
     clean_html,
+    dict_get,
     ExtractorError,
     float_or_none,
     int_or_none,
@@ -76,14 +77,14 @@ class TwitchBaseIE(InfoExtractor):
 
             headers = {
                 'Referer': page_url,
-                'Origin': page_url,
+                'Origin': 'https://www.twitch.tv',
                 'Content-Type': 'text/plain;charset=UTF-8',
             }
 
             response = self._download_json(
                 post_url, None, note, data=json.dumps(form).encode(),
                 headers=headers, expected_status=400)
-            error = response.get('error_description') or response.get('error_code')
+            error = dict_get(response, ('error', 'error_description', 'error_code'))
             if error:
                 fail(error)
 
@@ -137,13 +138,17 @@ class TwitchBaseIE(InfoExtractor):
         self._sort_formats(formats)
 
     def _download_base_gql(self, video_id, ops, note, fatal=True):
+        headers = {
+            'Content-Type': 'text/plain;charset=UTF-8',
+            'Client-ID': self._CLIENT_ID,
+        }
+        gql_auth = self._get_cookies('https://gql.twitch.tv').get('auth-token')
+        if gql_auth:
+            headers['Authorization'] = 'OAuth ' + gql_auth.value
         return self._download_json(
             'https://gql.twitch.tv/gql', video_id, note,
             data=json.dumps(ops).encode(),
-            headers={
-                'Content-Type': 'text/plain;charset=UTF-8',
-                'Client-ID': self._CLIENT_ID,
-            }, fatal=fatal)
+            headers=headers, fatal=fatal)
 
     def _download_gql(self, video_id, ops, note, fatal=True):
         for op in ops:

youtube_dlc/extractor/twitter.py

@@ -373,6 +373,24 @@ class TwitterIE(TwitterBaseIE):
             'uploader_id': '1eVjYOLGkGrQL',
         },
         'add_ie': ['TwitterBroadcast'],
+    }, {
+        # unified card
+        'url': 'https://twitter.com/BrooklynNets/status/1349794411333394432?s=20',
+        'info_dict': {
+            'id': '1349794411333394432',
+            'ext': 'mp4',
+            'title': 'md5:d1c4941658e4caaa6cb579260d85dcba',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'description': 'md5:71ead15ec44cee55071547d6447c6a3e',
+            'uploader': 'Brooklyn Nets',
+            'uploader_id': 'BrooklynNets',
+            'duration': 324.484,
+            'timestamp': 1610651040,
+            'upload_date': '20210114',
+        },
+        'params': {
+            'skip_download': True,
+        },
     }, {
         # Twitch Clip Embed
         'url': 'https://twitter.com/GunB1g/status/1163218564784017422',
@@ -389,6 +407,22 @@ class TwitterIE(TwitterBaseIE):
         # appplayer card
         'url': 'https://twitter.com/poco_dandy/status/1150646424461176832',
         'only_matching': True,
+    }, {
+        # video_direct_message card
+        'url': 'https://twitter.com/qarev001/status/1348948114569269251',
+        'only_matching': True,
+    }, {
+        # poll2choice_video card
+        'url': 'https://twitter.com/CAF_Online/status/1349365911120195585',
+        'only_matching': True,
+    }, {
+        # poll3choice_video card
+        'url': 'https://twitter.com/SamsungMobileSA/status/1348609186725289984',
+        'only_matching': True,
+    }, {
+        # poll4choice_video card
+        'url': 'https://twitter.com/SouthamptonFC/status/1347577658079641604',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -433,8 +467,7 @@ class TwitterIE(TwitterBaseIE):
             'tags': tags,
         }
 
-        media = try_get(status, lambda x: x['extended_entities']['media'][0])
-        if media and media.get('type') != 'photo':
+        def extract_from_video_info(media):
             video_info = media.get('video_info') or {}
 
             formats = []
@@ -461,6 +494,10 @@ class TwitterIE(TwitterBaseIE):
                 'thumbnails': thumbnails,
                 'duration': float_or_none(video_info.get('duration_millis'), 1000),
             })
+
+        media = try_get(status, lambda x: x['extended_entities']['media'][0])
+        if media and media.get('type') != 'photo':
+            extract_from_video_info(media)
         else:
             card = status.get('card')
             if card:
@@ -493,7 +530,12 @@ class TwitterIE(TwitterBaseIE):
                         '_type': 'url',
                         'url': get_binding_value('card_url'),
                     })
-                # amplify, promo_video_website, promo_video_convo, appplayer, ...
+                elif card_name == 'unified_card':
+                    media_entities = self._parse_json(get_binding_value('unified_card'), twid)['media_entities']
+                    extract_from_video_info(next(iter(media_entities.values())))
+                # amplify, promo_video_website, promo_video_convo, appplayer,
+                # video_direct_message, poll2choice_video, poll3choice_video,
+                # poll4choice_video, ...
                 else:
                     is_amplify = card_name == 'amplify'
                     vmap_url = get_binding_value('amplify_url_vmap') if is_amplify else get_binding_value('player_stream_url')

youtube_dlc/extractor/wat.py

@@ -1,12 +1,9 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
-
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
-    ExtractorError,
     unified_strdate,
     HEADRequest,
     int_or_none,
@@ -46,15 +43,6 @@ class WatIE(InfoExtractor):
         },
     ]
 
-    _FORMATS = (
-        (200, 416, 234),
-        (400, 480, 270),
-        (600, 640, 360),
-        (1200, 640, 360),
-        (1800, 960, 540),
-        (2500, 1280, 720),
-    )
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
         video_id = video_id if video_id.isdigit() and len(video_id) > 6 else compat_str(int(video_id, 36))
@@ -97,46 +85,20 @@ class WatIE(InfoExtractor):
                     return red_url
             return None
 
-        def remove_bitrate_limit(manifest_url):
-            return re.sub(r'(?:max|min)_bitrate=\d+&?', '', manifest_url)
-
         formats = []
-        try:
-            alt_urls = lambda manifest_url: [re.sub(r'(?:wdv|ssm)?\.ism/', repl + '.ism/', manifest_url) for repl in ('', 'ssm')]
-            manifest_urls = self._download_json(
-                'http://www.wat.tv/get/webhtml/' + video_id, video_id)
-            m3u8_url = manifest_urls.get('hls')
-            if m3u8_url:
-                m3u8_url = remove_bitrate_limit(m3u8_url)
-                for m3u8_alt_url in alt_urls(m3u8_url):
-                    formats.extend(self._extract_m3u8_formats(
-                        m3u8_alt_url, video_id, 'mp4',
-                        'm3u8_native', m3u8_id='hls', fatal=False))
-                    formats.extend(self._extract_f4m_formats(
-                        m3u8_alt_url.replace('ios', 'web').replace('.m3u8', '.f4m'),
-                        video_id, f4m_id='hds', fatal=False))
-            mpd_url = manifest_urls.get('mpd')
-            if mpd_url:
-                mpd_url = remove_bitrate_limit(mpd_url)
-                for mpd_alt_url in alt_urls(mpd_url):
-                    formats.extend(self._extract_mpd_formats(
-                        mpd_alt_url, video_id, mpd_id='dash', fatal=False))
-            self._sort_formats(formats)
-        except ExtractorError:
-            abr = 64
-            for vbr, width, height in self._FORMATS:
-                tbr = vbr + abr
-                format_id = 'http-%s' % tbr
-                fmt_url = 'http://dnl.adv.tf1.fr/2/USP-0x0/%s/%s/%s/ssm/%s-%s-64k.mp4' % (video_id[-4:-2], video_id[-2:], video_id, video_id, vbr)
-                if self._is_valid_url(fmt_url, video_id, format_id):
-                    formats.append({
-                        'format_id': format_id,
-                        'url': fmt_url,
-                        'vbr': vbr,
-                        'abr': abr,
-                        'width': width,
-                        'height': height,
-                    })
+        manifest_urls = self._download_json(
+            'http://www.wat.tv/get/webhtml/' + video_id, video_id)
+        m3u8_url = manifest_urls.get('hls')
+        if m3u8_url:
+            formats.extend(self._extract_m3u8_formats(
+                m3u8_url, video_id, 'mp4',
+                'm3u8_native', m3u8_id='hls', fatal=False))
+        mpd_url = manifest_urls.get('mpd')
+        if mpd_url:
+            formats.extend(self._extract_mpd_formats(
+                mpd_url.replace('://das-q1.tf1.fr/', '://das-q1-ssl.tf1.fr/'),
+                video_id, mpd_id='dash', fatal=False))
+        self._sort_formats(formats)
 
         date_diffusion = first_chapter.get('date_diffusion') or video_data.get('configv4', {}).get('estatS4')
         upload_date = unified_strdate(date_diffusion) if date_diffusion else None

youtube_dlc/extractor/yahoo.py

@@ -177,46 +177,9 @@ class YahooIE(InfoExtractor):
         'only_matching': True,
     }]
 
-    def _real_extract(self, url):
-        url, country, display_id = re.match(self._VALID_URL, url).groups()
-        if not country:
-            country = 'us'
-        else:
-            country = country.split('-')[0]
-        api_base = 'https://%s.yahoo.com/_td/api/resource/' % country
-
-        for i, uuid in enumerate(['url=' + url, 'ymedia-alias=' + display_id]):
-            content = self._download_json(
-                api_base + 'content;getDetailView=true;uuids=["%s"]' % uuid,
-                display_id, 'Downloading content JSON metadata', fatal=i == 1)
-            if content:
-                item = content['items'][0]
-                break
-
-        if item.get('type') != 'video':
-            entries = []
-
-            cover = item.get('cover') or {}
-            if cover.get('type') == 'yvideo':
-                cover_url = cover.get('url')
-                if cover_url:
-                    entries.append(self.url_result(
-                        cover_url, 'Yahoo', cover.get('uuid')))
-
-            for e in item.get('body', []):
-                if e.get('type') == 'videoIframe':
-                    iframe_url = e.get('url')
-                    if not iframe_url:
-                        continue
-                    entries.append(self.url_result(iframe_url))
-
-            return self.playlist_result(
-                entries, item.get('uuid'),
-                item.get('title'), item.get('summary'))
-
-        video_id = item['uuid']
+    def _extract_yahoo_video(self, video_id, country):
         video = self._download_json(
-            api_base + 'VideoService.videos;view=full;video_ids=["%s"]' % video_id,
+            'https://%s.yahoo.com/_td/api/resource/VideoService.videos;view=full;video_ids=["%s"]' % (country, video_id),
             video_id, 'Downloading video JSON metadata')[0]
         title = video['title']
 
@@ -298,7 +261,6 @@ class YahooIE(InfoExtractor):
             'id': video_id,
             'title': self._live_title(title) if is_live else title,
             'formats': formats,
-            'display_id': display_id,
             'thumbnails': thumbnails,
             'description': clean_html(video.get('description')),
             'timestamp': parse_iso8601(video.get('publish_time')),
@@ -311,6 +273,44 @@ class YahooIE(InfoExtractor):
             'episode_number': int_or_none(series_info.get('episode_number')),
         }
 
+    def _real_extract(self, url):
+        url, country, display_id = re.match(self._VALID_URL, url).groups()
+        if not country:
+            country = 'us'
+        else:
+            country = country.split('-')[0]
+
+        item = self._download_json(
+            'https://%s.yahoo.com/caas/content/article' % country, display_id,
+            'Downloading content JSON metadata', query={
+                'url': url
+            })['items'][0]['data']['partnerData']
+
+        if item.get('type') != 'video':
+            entries = []
+
+            cover = item.get('cover') or {}
+            if cover.get('type') == 'yvideo':
+                cover_url = cover.get('url')
+                if cover_url:
+                    entries.append(self.url_result(
+                        cover_url, 'Yahoo', cover.get('uuid')))
+
+            for e in (item.get('body') or []):
+                if e.get('type') == 'videoIframe':
+                    iframe_url = e.get('url')
+                    if not iframe_url:
+                        continue
+                    entries.append(self.url_result(iframe_url))
+
+            return self.playlist_result(
+                entries, item.get('uuid'),
+                item.get('title'), item.get('summary'))
+
+        info = self._extract_yahoo_video(item['uuid'], country)
+        info['display_id'] = display_id
+        return info
+
 
 class YahooSearchIE(SearchInfoExtractor):
     IE_DESC = 'Yahoo screen search'

youtube_dlc/extractor/youporn.py

@@ -60,6 +60,9 @@ class YouPornIE(InfoExtractor):
     }, {
         'url': 'http://www.youporn.com/watch/505835',
         'only_matching': True,
+    }, {
+        'url': 'https://www.youporn.com/watch/13922959/femdom-principal/',
+        'only_matching': True,
     }]
 
     @staticmethod
@@ -88,7 +91,7 @@ class YouPornIE(InfoExtractor):
         # Main source
         definitions = self._parse_json(
             self._search_regex(
-                r'mediaDefinition\s*=\s*(\[.+?\]);', webpage,
+                r'mediaDefinition\s*[=:]\s*(\[.+?\])\s*[;,]', webpage,
                 'media definitions', default='[]'),
             video_id, fatal=False)
         if definitions:
@@ -100,7 +103,7 @@ class YouPornIE(InfoExtractor):
                     links.append(video_url)
 
         # Fallback #1, this also contains extra low quality 180p format
-        for _, link in re.findall(r'<a[^>]+href=(["\'])(http.+?)\1[^>]+title=["\']Download [Vv]ideo', webpage):
+        for _, link in re.findall(r'<a[^>]+href=(["\'])(http(?:(?!\1).)+\.mp4(?:(?!\1).)*)\1[^>]+title=["\']Download [Vv]ideo', webpage):
             links.append(link)
 
         # Fallback #2 (unavailable as at 22.06.2017)
@@ -128,8 +131,9 @@ class YouPornIE(InfoExtractor):
             # Video URL's path looks like this:
             #  /201012/17/505835/720p_1500k_505835/YouPorn%20-%20Sex%20Ed%20Is%20It%20Safe%20To%20Masturbate%20Daily.mp4
             #  /201012/17/505835/vl_240p_240k_505835/YouPorn%20-%20Sex%20Ed%20Is%20It%20Safe%20To%20Masturbate%20Daily.mp4
+            #  /videos/201703/11/109285532/1080P_4000K_109285532.mp4
             # We will benefit from it by extracting some metadata
-            mobj = re.search(r'(?P<height>\d{3,4})[pP]_(?P<bitrate>\d+)[kK]_\d+/', video_url)
+            mobj = re.search(r'(?P<height>\d{3,4})[pP]_(?P<bitrate>\d+)[kK]_\d+', video_url)
             if mobj:
                 height = int(mobj.group('height'))
                 bitrate = int(mobj.group('bitrate'))

youtube_dlc/extractor/youtube.py

@@ -332,6 +332,36 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
                 r'ytcfg\.set\s*\(\s*({.+?})\s*\)\s*;', webpage, 'ytcfg',
                 default='{}'), video_id, fatal=False)
 
+    def _extract_video(self, renderer):
+        video_id = renderer.get('videoId')
+        title = try_get(
+            renderer,
+            (lambda x: x['title']['runs'][0]['text'],
+             lambda x: x['title']['simpleText']), compat_str)
+        description = try_get(
+            renderer, lambda x: x['descriptionSnippet']['runs'][0]['text'],
+            compat_str)
+        duration = parse_duration(try_get(
+            renderer, lambda x: x['lengthText']['simpleText'], compat_str))
+        view_count_text = try_get(
+            renderer, lambda x: x['viewCountText']['simpleText'], compat_str) or ''
+        view_count = str_to_int(self._search_regex(
+            r'^([\d,]+)', re.sub(r'\s', '', view_count_text),
+            'view count', default=None))
+        uploader = try_get(
+            renderer, lambda x: x['ownerText']['runs'][0]['text'], compat_str)
+        return {
+            '_type': 'url_transparent',
+            'ie_key': YoutubeIE.ie_key(),
+            'id': video_id,
+            'url': video_id,
+            'title': title,
+            'description': description,
+            'duration': duration,
+            'view_count': view_count,
+            'uploader': uploader,
+        }
+
 
 class YoutubeIE(YoutubeBaseInfoExtractor):
     IE_DESC = 'YouTube.com'
@@ -2394,9 +2424,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             default=None
         ))
 
-        # annotations
-        video_annotations = None
-        if self._downloader.params.get('writeannotations', False):
+        # get xsrf for annotations or comments
+        get_annotations = self._downloader.params.get('writeannotations', False)
+        get_comments = self._downloader.params.get('getcomments', False)
+        if get_annotations or get_comments:
             xsrf_token = None
             ytcfg = self._extract_ytcfg(video_id, video_webpage)
             if ytcfg:
@@ -2405,6 +2436,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 xsrf_token = self._search_regex(
                     r'([\'"])XSRF_TOKEN\1\s*:\s*([\'"])(?P<xsrf_token>(?:(?!\2).)+)\2',
                     video_webpage, 'xsrf token', group='xsrf_token', fatal=False)
+
+        # annotations
+        video_annotations = None
+        if get_annotations:
             invideo_url = try_get(
                 player_response, lambda x: x['annotations'][0]['playerAnnotationsUrlsRenderer']['invideoUrl'], compat_str)
             if xsrf_token and invideo_url:
@@ -2424,6 +2459,149 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
 
         chapters = self._extract_chapters(video_webpage, description_original, video_id, video_duration)
 
+        # Get comments
+        # TODO: Refactor and move to seperate function
+        if get_comments:
+            expected_video_comment_count = 0
+            video_comments = []
+
+            def find_value(html, key, num_chars=2, separator='"'):
+                pos_begin = html.find(key) + len(key) + num_chars
+                pos_end = html.find(separator, pos_begin)
+                return html[pos_begin: pos_end]
+
+            def search_dict(partial, key):
+                if isinstance(partial, dict):
+                    for k, v in partial.items():
+                        if k == key:
+                            yield v
+                        else:
+                            for o in search_dict(v, key):
+                                yield o
+                elif isinstance(partial, list):
+                    for i in partial:
+                        for o in search_dict(i, key):
+                            yield o
+
+            try:
+                ncd = next(search_dict(yt_initial_data, 'nextContinuationData'))
+                continuations = [ncd['continuation']]
+            # Handle videos where comments have been disabled entirely
+            except StopIteration:
+                continuations = []
+
+            def get_continuation(continuation, session_token, replies=False):
+                query = {
+                    'pbj': 1,
+                    'ctoken': continuation,
+                }
+                if replies:
+                    query['action_get_comment_replies'] = 1
+                else:
+                    query['action_get_comments'] = 1
+
+                while True:
+                    content, handle = self._download_webpage_handle(
+                        'https://www.youtube.com/comment_service_ajax',
+                        video_id,
+                        note=False,
+                        expected_status=[413],
+                        data=urlencode_postdata({
+                            'session_token': session_token
+                        }),
+                        query=query,
+                        headers={
+                            'Accept': '*/*',
+                            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0',
+                            'X-YouTube-Client-Name': '1',
+                            'X-YouTube-Client-Version': '2.20201202.06.01'
+                        }
+                    )
+
+                    response_code = handle.getcode()
+                    if (response_code == 200):
+                        return self._parse_json(content, video_id)
+                    if (response_code == 413):
+                        return None
+                    raise ExtractorError('Unexpected HTTP error code: %s' % response_code)
+
+            first_continuation = True
+            while continuations:
+                continuation, itct = continuations.pop()
+                comment_response = get_continuation(continuation, xsrf_token)
+                if not comment_response:
+                    continue
+                if list(search_dict(comment_response, 'externalErrorMessage')):
+                    raise ExtractorError('Error returned from server: ' + next(search_dict(comment_response, 'externalErrorMessage')))
+
+                if 'continuationContents' not in comment_response['response']:
+                    # Something is wrong here. Youtube won't accept this continuation token for some reason and responds with a user satisfaction dialog (error?)
+                    continue
+                # not sure if this actually helps
+                if 'xsrf_token' in comment_response:
+                    xsrf_token = comment_response['xsrf_token']
+
+                item_section = comment_response['response']['continuationContents']['itemSectionContinuation']
+                if first_continuation:
+                    expected_video_comment_count = int(item_section['header']['commentsHeaderRenderer']['countText']['runs'][0]['text'].replace(' Comments', '').replace('1 Comment', '1').replace(',', ''))
+                    first_continuation = False
+                if 'contents' not in item_section:
+                    # continuation returned no comments?
+                    # set an empty array as to not break the for loop
+                    item_section['contents'] = []
+
+                for meta_comment in item_section['contents']:
+                    comment = meta_comment['commentThreadRenderer']['comment']['commentRenderer']
+                    video_comments.append({
+                        'id': comment['commentId'],
+                        'text': ''.join([c['text'] for c in comment['contentText']['runs']]),
+                        'time_text': ''.join([c['text'] for c in comment['publishedTimeText']['runs']]),
+                        'author': comment.get('authorText', {}).get('simpleText', ''),
+                        'votes': comment.get('voteCount', {}).get('simpleText', '0'),
+                        'author_thumbnail': comment['authorThumbnail']['thumbnails'][-1]['url'],
+                        'parent': 'root'
+                    })
+                    if 'replies' not in meta_comment['commentThreadRenderer']:
+                        continue
+
+                    reply_continuations = [rcn['nextContinuationData']['continuation'] for rcn in meta_comment['commentThreadRenderer']['replies']['commentRepliesRenderer']['continuations']]
+                    while reply_continuations:
+                        time.sleep(1)
+                        continuation = reply_continuations.pop()
+                        replies_data = get_continuation(continuation, xsrf_token, True)
+                        if not replies_data or 'continuationContents' not in replies_data[1]['response']:
+                            continue
+
+                        if self._downloader.params.get('verbose', False):
+                            self.to_screen('[debug] Comments downloaded (chain %s) %s of ~%s' % (comment['commentId'], len(video_comments), expected_video_comment_count))
+                        reply_comment_meta = replies_data[1]['response']['continuationContents']['commentRepliesContinuation']
+                        for reply_meta in replies_data[1]['response']['continuationContents']['commentRepliesContinuation']['contents']:
+                            reply_comment = reply_meta['commentRenderer']
+                            video_comments.append({
+                                'id': reply_comment['commentId'],
+                                'text': ''.join([c['text'] for c in reply_comment['contentText']['runs']]),
+                                'time_text': ''.join([c['text'] for c in reply_comment['publishedTimeText']['runs']]),
+                                'author': reply_comment.get('authorText', {}).get('simpleText', ''),
+                                'votes': reply_comment.get('voteCount', {}).get('simpleText', '0'),
+                                'author_thumbnail': reply_comment['authorThumbnail']['thumbnails'][-1]['url'],
+                                'parent': comment['commentId']
+                            })
+                        if 'continuations' not in reply_comment_meta or len(reply_comment_meta['continuations']) == 0:
+                            continue
+
+                        reply_continuations += [rcn['nextContinuationData']['continuation'] for rcn in reply_comment_meta['continuations']]
+
+                self.to_screen('Comments downloaded %s of ~%s' % (len(video_comments), expected_video_comment_count))
+
+                if 'continuations' in item_section:
+                    continuations += [ncd['nextContinuationData']['continuation'] for ncd in item_section['continuations']]
+                time.sleep(1)
+
+            self.to_screen('Total comments downloaded %s of ~%s' % (len(video_comments), expected_video_comment_count))
+        else:
+            expected_video_comment_count = None
+            video_comments = None
+
         # Look for the DASH manifest
         if self._downloader.params.get('youtube_include_dash_manifest', True):
             dash_mpd_fatal = True
@@ -2542,6 +2720,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             'release_year': release_year,
             'subscriber_count': subscriber_count,
             'playable_in_embed': playable_in_embed,
+            'comments': video_comments,
+            'comment_count': expected_video_comment_count,
         }
 
 
@@ -2871,36 +3051,6 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             if renderer:
                 return renderer
 
-    def _extract_video(self, renderer):
-        video_id = renderer.get('videoId')
-        title = try_get(
-            renderer,
-            (lambda x: x['title']['runs'][0]['text'],
-             lambda x: x['title']['simpleText']), compat_str)
-        description = try_get(
-            renderer, lambda x: x['descriptionSnippet']['runs'][0]['text'],
-            compat_str)
-        duration = parse_duration(try_get(
-            renderer, lambda x: x['lengthText']['simpleText'], compat_str))
-        view_count_text = try_get(
-            renderer, lambda x: x['viewCountText']['simpleText'], compat_str) or ''
-        view_count = str_to_int(self._search_regex(
-            r'^([\d,]+)', re.sub(r'\s', '', view_count_text),
-            'view count', default=None))
-        uploader = try_get(
-            renderer, lambda x: x['ownerText']['runs'][0]['text'], compat_str)
-        return {
-            '_type': 'url_transparent',
-            'ie_key': YoutubeIE.ie_key(),
-            'id': video_id,
-            'url': video_id,
-            'title': title,
-            'description': description,
-            'duration': duration,
-            'view_count': view_count,
-            'uploader': uploader,
-        }
-
     def _grid_entries(self, grid_renderer):
         for item in grid_renderer['items']:
             if not isinstance(item, dict):
@@ -3276,12 +3426,19 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                 title += ' - %s' % tab_title
             description = renderer.get('description')
             playlist_id = renderer.get('externalId')
+
+        # this has thumbnails, but there is currently no thumbnail field for playlists
+        # sidebar.playlistSidebarRenderer has even more data, but its stucture is more complec
         renderer = try_get(
-            data, lambda x: x['metadata']['playlistMetadataRenderer'], dict)
+            data, lambda x: x['microformat']['microformatDataRenderer'], dict)
+        if not renderer:
+            renderer = try_get(
+                data, lambda x: x['metadata']['playlistMetadataRenderer'], dict)
         if renderer:
             title = renderer.get('title')
-            description = None
+            description = renderer.get('description')
             playlist_id = item_id
+
         if playlist_id is None:
             playlist_id = item_id
         if title is None:
@@ -3371,8 +3528,16 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
         webpage = self._download_webpage(url, item_id)
         identity_token = self._extract_identity_token(webpage, item_id)
         data = self._extract_yt_initial_data(item_id, webpage)
+        err_msg = None
         for alert_type, alert_message in self._extract_alerts(data):
-            self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
+            if alert_type.lower() == 'error':
+                if err_msg:
+                    self._downloader.report_warning('YouTube said: %s - %s' % ('ERROR', err_msg))
+                err_msg = alert_message
+            else:
+                self._downloader.report_warning('YouTube said: %s - %s' % (alert_type, alert_message))
+        if err_msg:
+            raise ExtractorError('YouTube said: %s' % err_msg, expected=True)
         tabs = try_get(
             data, lambda x: x['contents']['twoColumnBrowseResultsRenderer']['tabs'], list)
         if tabs:
@@ -3583,65 +3748,38 @@ class YoutubeSearchIE(SearchInfoExtractor, YoutubeBaseInfoExtractor):
             if not slr_contents:
                 break
 
-            isr_contents = []
-            continuation_token = None
             # Youtube sometimes adds promoted content to searches,
             # changing the index location of videos and token.
             # So we search through all entries till we find them.
-            for index, isr in enumerate(slr_contents):
+            continuation_token = None
+            for slr_content in slr_contents:
+                isr_contents = try_get(
+                    slr_content,
+                    lambda x: x['itemSectionRenderer']['contents'],
+                    list)
                 if not isr_contents:
-                    isr_contents = try_get(
-                        slr_contents,
-                        (lambda x: x[index]['itemSectionRenderer']['contents']),
-                        list)
-                    for content in isr_contents:
-                        if content.get('videoRenderer') is not None:
-                            break
-                    else:
-                        isr_contents = []
+                    continue
+                for content in isr_contents:
+                    if not isinstance(content, dict):
+                        continue
+                    video = content.get('videoRenderer')
+                    if not isinstance(video, dict):
+                        continue
+                    video_id = video.get('videoId')
+                    if not video_id:
+                        continue
+
+                    yield self._extract_video(video)
+                    total += 1
+                    if total == n:
+                        return
 
                 if continuation_token is None:
                     continuation_token = try_get(
-                        slr_contents,
-                        lambda x: x[index]['continuationItemRenderer']['continuationEndpoint']['continuationCommand'][
-                            'token'],
+                        slr_content,
+                        lambda x: x['continuationItemRenderer']['continuationEndpoint']['continuationCommand']['token'],
                         compat_str)
-                if continuation_token is not None and isr_contents:
-                    break
 
-            if not isr_contents:
-                break
-            for content in isr_contents:
-                if not isinstance(content, dict):
-                    continue
-                video = content.get('videoRenderer')
-                if not isinstance(video, dict):
-                    continue
-                video_id = video.get('videoId')
-                if not video_id:
-                    continue
-                title = try_get(video, lambda x: x['title']['runs'][0]['text'], compat_str)
-                description = try_get(video, lambda x: x['descriptionSnippet']['runs'][0]['text'], compat_str)
-                duration = parse_duration(try_get(video, lambda x: x['lengthText']['simpleText'], compat_str))
-                view_count_text = try_get(video, lambda x: x['viewCountText']['simpleText'], compat_str) or ''
-                view_count = str_to_int(self._search_regex(
-                    r'^([\d,]+)', re.sub(r'\s', '', view_count_text),
-                    'view count', default=None))
-                uploader = try_get(video, lambda x: x['ownerText']['runs'][0]['text'], compat_str)
-                total += 1
-                yield {
-                    '_type': 'url_transparent',
-                    'ie_key': YoutubeIE.ie_key(),
-                    'id': video_id,
-                    'url': video_id,
-                    'title': title,
-                    'description': description,
-                    'duration': duration,
-                    'view_count': view_count,
-                    'uploader': uploader,
-                }
-                if total == n:
-                    return
             if not continuation_token:
                 break
             data['continuation'] = continuation_token

youtube_dlc/options.py

@@ -14,12 +14,17 @@ from .compat import (
     compat_shlex_split,
 )
 from .utils import (
+    expand_path,
+    get_executable_path,
     preferredencoding,
     write_string,
 )
 from .version import __version__
 
 
+_remux_formats = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
+
+
 def _hide_login_info(opts):
     PRIVATE_OPTS = set(['-p', '--password', '-u', '--username', '--video-password', '--ap-password', '--ap-username'])
     eqre = re.compile('^(?P<key>' + ('|'.join(re.escape(po) for po in PRIVATE_OPTS)) + ')=.+$')
@@ -54,42 +59,37 @@ def parseOpts(overrideArguments=None):
             optionf.close()
         return res
 
-    def _readUserConf():
-        xdg_config_home = compat_getenv('XDG_CONFIG_HOME')
-        if xdg_config_home:
-            userConfFile = os.path.join(xdg_config_home, 'youtube-dlc', 'config')
-            if not os.path.isfile(userConfFile):
-                userConfFile = os.path.join(xdg_config_home, 'youtube-dlc.conf')
-        else:
-            userConfFile = os.path.join(compat_expanduser('~'), '.config', 'youtube-dlc', 'config')
-            if not os.path.isfile(userConfFile):
-                userConfFile = os.path.join(compat_expanduser('~'), '.config', 'youtube-dlc.conf')
-        userConf = _readOptions(userConfFile, None)
+    def _readUserConf(package_name, default=[]):
+        # .config
+        xdg_config_home = compat_getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config')
+        userConfFile = os.path.join(xdg_config_home, package_name, 'config')
+        if not os.path.isfile(userConfFile):
+            userConfFile = os.path.join(xdg_config_home, '%s.conf' % package_name)
+        userConf = _readOptions(userConfFile, default=None)
+        if userConf is not None:
+            return userConf, userConfFile
 
-        if userConf is None:
-            appdata_dir = compat_getenv('appdata')
-            if appdata_dir:
-                userConf = _readOptions(
-                    os.path.join(appdata_dir, 'youtube-dlc', 'config'),
-                    default=None)
-                if userConf is None:
-                    userConf = _readOptions(
-                        os.path.join(appdata_dir, 'youtube-dlc', 'config.txt'),
-                        default=None)
+        # appdata
+        appdata_dir = compat_getenv('appdata')
+        if appdata_dir:
+            userConfFile = os.path.join(appdata_dir, package_name, 'config')
+            userConf = _readOptions(userConfFile, default=None)
+            if userConf is None:
+                userConfFile += '.txt'
+                userConf = _readOptions(userConfFile, default=None)
+        if userConf is not None:
+            return userConf, userConfFile
 
+        # home
+        userConfFile = os.path.join(compat_expanduser('~'), '%s.conf' % package_name)
+        userConf = _readOptions(userConfFile, default=None)
         if userConf is None:
-            userConf = _readOptions(
-                os.path.join(compat_expanduser('~'), 'youtube-dlc.conf'),
-                default=None)
-        if userConf is None:
-            userConf = _readOptions(
-                os.path.join(compat_expanduser('~'), 'youtube-dlc.conf.txt'),
-                default=None)
+            userConfFile += '.txt'
+            userConf = _readOptions(userConfFile, default=None)
+        if userConf is not None:
+            return userConf, userConfFile
 
-        if userConf is None:
-            userConf = []
-
-        return userConf
+        return default, None
 
     def _format_option_string(option):
         ''' ('-o', '--option') -> -o, --format METAVAR'''
@@ -111,6 +111,20 @@ def parseOpts(overrideArguments=None):
     def _comma_separated_values_options_callback(option, opt_str, value, parser):
         setattr(parser.values, option.dest, value.split(','))
 
+    def _dict_from_multiple_values_options_callback(
+            option, opt_str, value, parser, allowed_keys=r'[\w-]+', delimiter=':', default_key=None, process=None):
+
+        out_dict = getattr(parser.values, option.dest)
+        mobj = re.match(r'(?i)(?P<key>%s)%s(?P<val>.*)$' % (allowed_keys, delimiter), value)
+        if mobj is not None:
+            key, val = mobj.group('key').lower(), mobj.group('val')
+        elif default_key is not None:
+            key, val = default_key, value
+        else:
+            raise optparse.OptionValueError(
+                'wrong %s formatting; it should be %s, not "%s"' % (opt_str, option.metavar, value))
+        out_dict[key] = process(val) if callable(process) else val
+
     # No need to wrap help messages if we're on a wide console
     columns = compat_get_terminal_size().columns
     max_width = columns if columns else 80
@@ -144,11 +158,11 @@ def parseOpts(overrideArguments=None):
     general.add_option(
         '-i', '--ignore-errors', '--no-abort-on-error',
         action='store_true', dest='ignoreerrors', default=True,
-        help='Continue on download errors, for example to skip unavailable videos in a playlist (default)')
+        help='Continue on download errors, for example to skip unavailable videos in a playlist (default) (Alias: --no-abort-on-error)')
     general.add_option(
         '--abort-on-error', '--no-ignore-errors',
         action='store_false', dest='ignoreerrors',
-        help='Abort downloading of further videos if an error occurs')
+        help='Abort downloading of further videos if an error occurs (Alias: --no-ignore-errors)')
     general.add_option(
         '--dump-user-agent',
         action='store_true', dest='dump_user_agent', default=False,
@@ -168,23 +182,23 @@ def parseOpts(overrideArguments=None):
     general.add_option(
         '--default-search',
         dest='default_search', metavar='PREFIX',
-        help='Use this prefix for unqualified URLs. For example "gvsearch2:" downloads two videos from google videos for youtube-dl "large apple". Use the value "auto" to let youtube-dl guess ("auto_warning" to emit a warning when guessing). "error" just throws an error. The default value "fixup_error" repairs broken URLs, but emits an error if this is not possible instead of searching.')
+        help='Use this prefix for unqualified URLs. For example "gvsearch2:" downloads two videos from google videos for youtube-dl "large apple". Use the value "auto" to let youtube-dl guess ("auto_warning" to emit a warning when guessing). "error" just throws an error. The default value "fixup_error" repairs broken URLs, but emits an error if this is not possible instead of searching')
     general.add_option(
         '--ignore-config', '--no-config',
         action='store_true',
         help=(
-            'Do not read configuration files. '
-            'When given in the global configuration file /etc/youtube-dl.conf: '
-            'Do not read the user configuration in ~/.config/youtube-dl/config '
-            '(%APPDATA%/youtube-dl/config.txt on Windows)'))
+            'Disable loading any configuration files except the one provided by --config-location. '
+            'When given inside a configuration file, no further configuration files are loaded. '
+            'Additionally, (for backward compatibility) if this option is found inside the '
+            'system configuration file, the user configuration is not loaded'))
     general.add_option(
         '--config-location',
         dest='config_location', metavar='PATH',
-        help='Location of the configuration file; either the path to the config or its containing directory.')
+        help='Location of the main configuration file; either the path to the config or its containing directory')
     general.add_option(
         '--flat-playlist',
         action='store_const', dest='extract_flat', const='in_playlist', default=False,
-        help='Do not extract the videos of a playlist, only list them.')
+        help='Do not extract the videos of a playlist, only list them')
     general.add_option(
         '--flat-videos',
         action='store_true', dest='extract_flat',
@@ -202,7 +216,7 @@ def parseOpts(overrideArguments=None):
         action='store_false', dest='mark_watched', default=False,
         help='Do not mark videos watched')
     general.add_option(
-        '--no-color', '--no-colors',
+        '--no-colors',
         action='store_true', dest='no_color',
         default=False,
         help='Do not emit color codes in output')
@@ -242,7 +256,7 @@ def parseOpts(overrideArguments=None):
         dest='geo_verification_proxy', default=None, metavar='URL',
         help=(
             'Use this proxy to verify the IP address for some geo-restricted sites. '
-            'The default proxy specified by --proxy (or none, if the option is not present) is used for the actual downloading.'))
+            'The default proxy specified by --proxy (or none, if the option is not present) is used for the actual downloading'))
     geo.add_option(
         '--cn-verification-proxy',
         dest='cn_verification_proxy', default=None, metavar='URL',
@@ -276,7 +290,7 @@ def parseOpts(overrideArguments=None):
     selection.add_option(
         '--playlist-items',
         dest='playlist_items', metavar='ITEM_SPEC', default=None,
-        help='Playlist video items to download. Specify indices of the videos in the playlist separated by commas like: "--playlist-items 1,2,5,8" if you want to download videos indexed 1, 2, 5, 8 in the playlist. You can specify range: "--playlist-items 1-3,7,10-13", it will download the videos at index 1, 2, 3, 7, 10, 11, 12 and 13.')
+        help='Playlist video items to download. Specify indices of the videos in the playlist separated by commas like: "--playlist-items 1,2,5,8" if you want to download videos indexed 1, 2, 5, 8 in the playlist. You can specify range: "--playlist-items 1-3,7,10-13", it will download the videos at index 1, 2, 3, 7, 10, 11, 12 and 13')
     selection.add_option(
         '--match-title',
         dest='matchtitle', metavar='REGEX',
@@ -301,8 +315,8 @@ def parseOpts(overrideArguments=None):
         '--date',
         metavar='DATE', dest='date', default=None,
         help=(
-            'Download only videos uploaded in this date.'
-            'The date can be "YYYYMMDD" or in the format'
+            'Download only videos uploaded in this date. '
+            'The date can be "YYYYMMDD" or in the format '
             '"(now|today)[+-][0-9](day|week|month|year)(s)?"'))
     selection.add_option(
         '--datebefore',
@@ -329,10 +343,10 @@ def parseOpts(overrideArguments=None):
         metavar='FILTER', dest='match_filter', default=None,
         help=(
             'Generic video filter. '
-            'Specify any key (see the "OUTPUT TEMPLATE" for a list of available keys) to '
+            'Specify any key (see "OUTPUT TEMPLATE" for a list of available keys) to '
             'match if the key is present, '
             '!key to check if the key is not present, '
-            'key > NUMBER (like "comment_count > 12", also works with '
+            'key>NUMBER (like "comment_count > 12", also works with '
             '>=, <, <=, !=, =) to compare against a number, '
             'key = \'LITERAL\' (like "uploader = \'Mike Smith\'", also works with !=) '
             'to match against a string literal '
@@ -343,7 +357,7 @@ def parseOpts(overrideArguments=None):
             '100 times and disliked less than 50 times (or the dislike '
             'functionality is not available at the given service), but who '
             'also have a description, use --match-filter '
-            '"like_count > 100 & dislike_count <? 50 & description" .'))
+            '"like_count > 100 & dislike_count <? 50 & description"'))
     selection.add_option(
         '--no-match-filter',
         metavar='FILTER', dest='match_filter', action='store_const', const=None,
@@ -351,11 +365,11 @@ def parseOpts(overrideArguments=None):
     selection.add_option(
         '--no-playlist',
         action='store_true', dest='noplaylist', default=False,
-        help='Download only the video, if the URL refers to a video and a playlist.')
+        help='Download only the video, if the URL refers to a video and a playlist')
     selection.add_option(
         '--yes-playlist',
         action='store_false', dest='noplaylist', default=False,
-        help='Download the playlist, if the URL refers to a video and a playlist.')
+        help='Download the playlist, if the URL refers to a video and a playlist')
     selection.add_option(
         '--age-limit',
         metavar='YEARS', dest='age_limit', default=None, type=int,
@@ -363,11 +377,15 @@ def parseOpts(overrideArguments=None):
     selection.add_option(
         '--download-archive', metavar='FILE',
         dest='download_archive',
-        help='Download only videos not listed in the archive file. Record the IDs of all downloaded videos in it.')
+        help='Download only videos not listed in the archive file. Record the IDs of all downloaded videos in it')
     selection.add_option(
         '--break-on-existing',
         action='store_true', dest='break_on_existing', default=False,
-        help="Stop the download process after attempting to download a file that's in the archive.")
+        help='Stop the download process when encountering a file that is in the archive')
+    selection.add_option(
+        '--break-on-reject',
+        action='store_true', dest='break_on_reject', default=False,
+        help='Stop the download process when encountering a file that has been filtered out')
     selection.add_option(
         '--no-download-archive',
         dest='download_archive', action="store_const", const=None,
@@ -389,7 +407,7 @@ def parseOpts(overrideArguments=None):
     authentication.add_option(
         '-p', '--password',
         dest='password', metavar='PASSWORD',
-        help='Account password. If this option is left out, youtube-dlc will ask interactively.')
+        help='Account password. If this option is left out, youtube-dlc will ask interactively')
     authentication.add_option(
         '-2', '--twofactor',
         dest='twofactor', metavar='TWOFACTOR',
@@ -415,7 +433,7 @@ def parseOpts(overrideArguments=None):
     adobe_pass.add_option(
         '--ap-password',
         dest='ap_password', metavar='PASSWORD',
-        help='Multiple-system operator account password. If this option is left out, youtube-dlc will ask interactively.')
+        help='Multiple-system operator account password. If this option is left out, youtube-dlc will ask interactively')
     adobe_pass.add_option(
         '--ap-list-mso',
         action='store_true', dest='ap_list_mso', default=False,
@@ -474,27 +492,27 @@ def parseOpts(overrideArguments=None):
     video_format.add_option(
         '--list-formats-as-table',
         action='store_true', dest='listformats_table', default=True,
-        help='Present the output of -F in a more tabular form (default)')
+        help='Present the output of -F in tabular form (default)')
     video_format.add_option(
         '--list-formats-old', '--no-list-formats-as-table',
         action='store_false', dest='listformats_table',
-        help='Present the output of -F in the old form')
+        help='Present the output of -F in the old form (Alias: --no-list-formats-as-table)')
     video_format.add_option(
         '--youtube-include-dash-manifest', '--no-youtube-skip-dash-manifest',
         action='store_true', dest='youtube_include_dash_manifest', default=True,
-        help='Download the DASH manifests and related data on YouTube videos (default)')
+        help='Download the DASH manifests and related data on YouTube videos (default) (Alias: --no-youtube-skip-dash-manifest)')
     video_format.add_option(
         '--youtube-skip-dash-manifest', '--no-youtube-include-dash-manifest',
         action='store_false', dest='youtube_include_dash_manifest',
-        help='Do not download the DASH manifests and related data on YouTube videos')
+        help='Do not download the DASH manifests and related data on YouTube videos (Alias: --no-youtube-include-dash-manifest)')
     video_format.add_option(
         '--youtube-include-hls-manifest', '--no-youtube-skip-hls-manifest',
         action='store_true', dest='youtube_include_hls_manifest', default=True,
-        help='Download the HLS manifests and related data on YouTube videos (default)')
+        help='Download the HLS manifests and related data on YouTube videos (default) (Alias: --no-youtube-skip-hls-manifest)')
     video_format.add_option(
         '--youtube-skip-hls-manifest', '--no-youtube-include-hls-manifest',
         action='store_false', dest='youtube_include_hls_manifest',
-        help='Do not download the HLS manifests and related data on YouTube videos')
+        help='Do not download the HLS manifests and related data on YouTube videos (Alias: --no-youtube-include-hls-manifest)')
     video_format.add_option(
         '--merge-output-format',
         action='store', dest='merge_output_format', metavar='FORMAT', default=None,
@@ -546,7 +564,7 @@ def parseOpts(overrideArguments=None):
     downloader.add_option(
         '-R', '--retries',
         dest='retries', metavar='RETRIES', default=10,
-        help='Number of retries (default is %default), or "infinite".')
+        help='Number of retries (default is %default), or "infinite"')
     downloader.add_option(
         '--fragment-retries',
         dest='fragment_retries', metavar='RETRIES', default=10,
@@ -554,11 +572,11 @@ def parseOpts(overrideArguments=None):
     downloader.add_option(
         '--skip-unavailable-fragments', '--no-abort-on-unavailable-fragment',
         action='store_true', dest='skip_unavailable_fragments', default=True,
-        help='Skip unavailable fragments for DASH, hlsnative and ISM (default)')
+        help='Skip unavailable fragments for DASH, hlsnative and ISM (default) (Alias: --no-abort-on-unavailable-fragment)')
     downloader.add_option(
         '--abort-on-unavailable-fragment', '--no-skip-unavailable-fragments',
         action='store_false', dest='skip_unavailable_fragments',
-        help='Abort downloading when some fragment is unavailable')
+        help='Abort downloading if a fragment is unavailable (Alias: --no-skip-unavailable-fragments)')
     downloader.add_option(
         '--keep-fragments',
         action='store_true', dest='keep_fragments', default=False,
@@ -621,14 +639,21 @@ def parseOpts(overrideArguments=None):
             'video while downloading (some players may not be able to play it)'))
     downloader.add_option(
         '--external-downloader',
-        dest='external_downloader', metavar='COMMAND',
+        dest='external_downloader', metavar='NAME',
         help=(
             'Use the specified external downloader. '
-            'Currently supports %s' % ','.join(list_external_downloaders())))
+            'Currently supports %s' % ', '.join(list_external_downloaders())))
     downloader.add_option(
-        '--external-downloader-args',
-        dest='external_downloader_args', metavar='ARGS',
-        help='Give these arguments to the external downloader')
+        '--downloader-args', '--external-downloader-args',
+        metavar='NAME:ARGS', dest='external_downloader_args', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={
+            'allowed_keys': '|'.join(list_external_downloaders()),
+            'default_key': 'default', 'process': compat_shlex_split},
+        help=(
+            'Give these arguments to the external downloader. '
+            'Specify the downloader name and the arguments separated by a colon ":". '
+            'You can use this option multiple times (Alias: --external-downloader-args)'))
 
     workarounds = optparse.OptionGroup(parser, 'Workarounds')
     workarounds.add_option(
@@ -654,8 +679,9 @@ def parseOpts(overrideArguments=None):
     )
     workarounds.add_option(
         '--add-header',
-        metavar='FIELD:VALUE', dest='headers', action='append',
-        help='Specify a custom HTTP header and its value, separated by a colon \':\'. You can use this option multiple times',
+        metavar='FIELD:VALUE', dest='headers', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        help='Specify a custom HTTP header and its value, separated by a colon ":". You can use this option multiple times',
     )
     workarounds.add_option(
         '--bidi-workaround',
@@ -668,14 +694,14 @@ def parseOpts(overrideArguments=None):
             'Number of seconds to sleep before each download when used alone '
             'or a lower bound of a range for randomized sleep before each download '
             '(minimum possible number of seconds to sleep) when used along with '
-            '--max-sleep-interval.'))
+            '--max-sleep-interval'))
     workarounds.add_option(
         '--max-sleep-interval', metavar='SECONDS',
         dest='max_sleep_interval', type=float,
         help=(
             'Upper bound of a range for randomized sleep before each download '
             '(maximum possible number of seconds to sleep). Must only be used '
-            'along with --min-sleep-interval.'))
+            'along with --min-sleep-interval'))
     workarounds.add_option(
         '--sleep-subtitles', metavar='SECONDS',
         dest='sleep_interval_subtitles', default=0, type=int,
@@ -733,23 +759,23 @@ def parseOpts(overrideArguments=None):
     verbosity.add_option(
         '-j', '--dump-json',
         action='store_true', dest='dumpjson', default=False,
-        help='Simulate, quiet but print JSON information. See the "OUTPUT TEMPLATE" for a description of available keys.')
+        help='Simulate, quiet but print JSON information. See "OUTPUT TEMPLATE" for a description of available keys')
     verbosity.add_option(
         '-J', '--dump-single-json',
         action='store_true', dest='dump_single_json', default=False,
         help=(
             'Simulate, quiet but print JSON information for each command-line argument. '
-            'If the URL refers to a playlist, dump the whole playlist information in a single line.'))
+            'If the URL refers to a playlist, dump the whole playlist information in a single line'))
     verbosity.add_option(
         '--print-json',
         action='store_true', dest='print_json', default=False,
-        help='Be quiet and print the video information as JSON (video is still being downloaded).')
+        help='Be quiet and print the video information as JSON (video is still being downloaded)')
     verbosity.add_option(
         '--force-write-archive', '--force-write-download-archive', '--force-download-archive',
         action='store_true', dest='force_write_download_archive', default=False,
         help=(
             'Force download archive entries to be written as far as no errors occur,'
-            'even if -s or another simulation switch is used.'))
+            'even if -s or another simulation switch is used (Alias: --force-download-archive)'))
     verbosity.add_option(
         '--newline',
         action='store_true', dest='progress_with_newline', default=False,
@@ -796,14 +822,33 @@ def parseOpts(overrideArguments=None):
         '-a', '--batch-file',
         dest='batchfile', metavar='FILE',
         help="File containing URLs to download ('-' for stdin), one URL per line. "
-             "Lines starting with '#', ';' or ']' are considered as comments and ignored.")
+             "Lines starting with '#', ';' or ']' are considered as comments and ignored")
     filesystem.add_option(
         '--id', default=False,
         action='store_true', dest='useid', help=optparse.SUPPRESS_HELP)
+    filesystem.add_option(
+        '-P', '--paths',
+        metavar='TYPE:PATH', dest='paths', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={
+            'allowed_keys': 'home|temp|config|description|annotation|subtitle|infojson|thumbnail',
+            'process': lambda x: x.strip()},
+        help=(
+            'The paths where the files should be downloaded. '
+            'Specify the type of file and the path separated by a colon ":" '
+            '(supported: description|annotation|subtitle|infojson|thumbnail). '
+            'Additionally, you can also provide "home" and "temp" paths. '
+            'All intermediary files are first downloaded to the temp path and '
+            'then the final files are moved over to the home path after download is finished. '
+            'Note that this option is ignored if --output is an absolute path'))
     filesystem.add_option(
         '-o', '--output',
         dest='outtmpl', metavar='TEMPLATE',
-        help='Output filename template, see the "OUTPUT TEMPLATE" for details')
+        help='Output filename template, see "OUTPUT TEMPLATE" for details')
+    filesystem.add_option(
+        '--output-na-placeholder',
+        dest='outtmpl_na_placeholder', metavar='TEXT', default='NA',
+        help=('Placeholder value for unavailable meta fields in output filename template (default: "%default")'))
     filesystem.add_option(
         '--autonumber-size',
         dest='autonumber_size', metavar='NUMBER', type=int,
@@ -892,6 +937,10 @@ def parseOpts(overrideArguments=None):
         '--no-write-annotations',
         action='store_false', dest='writeannotations',
         help='Do not write video annotations (default)')
+    filesystem.add_option(
+        '--get-comments',
+        action='store_true', dest='getcomments', default=False,
+        help='Retrieve video comments to be placed in the .info.json file')
     filesystem.add_option(
         '--load-info-json', '--load-info',
         dest='load_info_filename', metavar='FILE',
@@ -906,7 +955,7 @@ def parseOpts(overrideArguments=None):
         help='Do not read/dump cookies (default)')
     filesystem.add_option(
         '--cache-dir', dest='cachedir', default=None, metavar='DIR',
-        help='Location in the filesystem where youtube-dl can store some downloaded information permanently. By default $XDG_CACHE_HOME/youtube-dl or ~/.cache/youtube-dl . At the moment, only YouTube player files (for videos with obfuscated signatures) are cached, but that may change.')
+        help='Location in the filesystem where youtube-dl can store some downloaded information permanently. By default $XDG_CACHE_HOME/youtube-dl or ~/.cache/youtube-dl . At the moment, only YouTube player files (for videos with obfuscated signatures) are cached, but that may change')
     filesystem.add_option(
         '--no-cache-dir', action='store_false', dest='cachedir',
         help='Disable filesystem caching')
@@ -941,51 +990,62 @@ def parseOpts(overrideArguments=None):
     link.add_option(
         '--write-link',
         action='store_true', dest='writelink', default=False,
-        help='Write an internet shortcut file, depending on the current platform (.url/.webloc/.desktop). The URL may be cached by the OS.')
+        help='Write an internet shortcut file, depending on the current platform (.url, .webloc or .desktop). The URL may be cached by the OS')
     link.add_option(
         '--write-url-link',
         action='store_true', dest='writeurllink', default=False,
-        help='Write a Windows internet shortcut file (.url). Note that the OS caches the URL based on the file path.')
+        help='Write a .url Windows internet shortcut. The OS caches the URL based on the file path')
     link.add_option(
         '--write-webloc-link',
         action='store_true', dest='writewebloclink', default=False,
-        help='Write a macOS internet shortcut file (.webloc)')
+        help='Write a .webloc macOS internet shortcut')
     link.add_option(
         '--write-desktop-link',
         action='store_true', dest='writedesktoplink', default=False,
-        help='Write a Linux internet shortcut file (.desktop)')
+        help='Write a .desktop Linux internet shortcut')
 
     postproc = optparse.OptionGroup(parser, 'Post-Processing Options')
     postproc.add_option(
         '-x', '--extract-audio',
         action='store_true', dest='extractaudio', default=False,
-        help='Convert video files to audio-only files (requires ffmpeg or avconv and ffprobe or avprobe)')
+        help='Convert video files to audio-only files (requires ffmpeg and ffprobe)')
     postproc.add_option(
         '--audio-format', metavar='FORMAT', dest='audioformat', default='best',
         help='Specify audio format: "best", "aac", "flac", "mp3", "m4a", "opus", "vorbis", or "wav"; "%default" by default; No effect without -x')
     postproc.add_option(
         '--audio-quality', metavar='QUALITY',
         dest='audioquality', default='5',
-        help='Specify ffmpeg/avconv audio quality, insert a value between 0 (better) and 9 (worse) for VBR or a specific bitrate like 128K (default %default)')
+        help='Specify ffmpeg audio quality, insert a value between 0 (better) and 9 (worse) for VBR or a specific bitrate like 128K (default %default)')
     postproc.add_option(
         '--remux-video',
         metavar='FORMAT', dest='remuxvideo', default=None,
         help=(
-            'Remux the video into another container if necessary (currently supported: mp4|mkv). '
-            'If target container does not support the video/audio codec, remuxing will fail'))
+            'Remux the video into another container if necessary (currently supported: %s). '
+            'If target container does not support the video/audio codec, remuxing will fail. '
+            'You can specify multiple rules; eg. "aac>m4a/mov>mp4/mkv" will remux aac to m4a, mov to mp4 '
+            'and anything else to mkv.' % '|'.join(_remux_formats)))
     postproc.add_option(
         '--recode-video',
         metavar='FORMAT', dest='recodevideo', default=None,
-        help='Re-encode the video into another format if re-encoding is necessary (currently supported: mp4|flv|ogg|webm|mkv|avi)')
+        help=(
+            'Re-encode the video into another format if re-encoding is necessary. '
+            'The supported formats are the same as --remux-video'))
     postproc.add_option(
-        '--postprocessor-args', metavar='NAME:ARGS',
-        dest='postprocessor_args', action='append',
+        '--postprocessor-args', '--ppa',
+        metavar='NAME:ARGS', dest='postprocessor_args', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={'default_key': 'default-compat', 'allowed_keys': r'\w+(?:\+\w+)?', 'process': compat_shlex_split},
         help=(
             'Give these arguments to the postprocessors. '
-            "Specify the postprocessor name and the arguments separated by a colon ':' "
-            'to give the argument to only the specified postprocessor. Supported names are '
-            'ExtractAudio, VideoRemuxer, VideoConvertor, EmbedSubtitle, Metadata, Merger, FixupStretched, FixupM4a, FixupM3u8, SubtitlesConvertor, SponSkrub and Default'
-            '. You can use this option multiple times to give different arguments to different postprocessors'))
+            'Specify the postprocessor/executable name and the arguments separated by a colon ":" '
+            'to give the argument to the specified postprocessor/executable. Supported postprocessors are: '
+            'SponSkrub, ExtractAudio, VideoRemuxer, VideoConvertor, EmbedSubtitle, Metadata, Merger, '
+            'FixupStretched, FixupM4a, FixupM3u8, SubtitlesConvertor and EmbedThumbnail. '
+            'The supported executables are: SponSkrub, FFmpeg, FFprobe, and AtomicParsley. '
+            'You can use this option multiple times to give different arguments to different postprocessors. '
+            'You can also specify "PP+EXE:ARGS" to give the arguments to the specified executable '
+            'only when being used by the specified postprocessor. '
+            'You can use this option multiple times (Alias: --ppa)'))
     postproc.add_option(
         '-k', '--keep-video',
         action='store_true', dest='keepvideo', default=False,
@@ -1029,14 +1089,20 @@ def parseOpts(overrideArguments=None):
     postproc.add_option(
         '--metadata-from-title',
         metavar='FORMAT', dest='metafromtitle',
+        help=optparse.SUPPRESS_HELP)
+    postproc.add_option(
+        '--parse-metadata',
+        metavar='FIELD:FORMAT', dest='metafromfield', action='append',
         help=(
-            'Parse additional metadata like song title / artist from the video title. '
-            'The format syntax is the same as --output. Regular expression with '
-            'named capture groups may also be used. '
-            'The parsed parameters replace existing values. '
-            'Example: --metadata-from-title "%(artist)s - %(title)s" matches a title like '
+            'Parse additional metadata like title/artist from other fields. '
+            'Give field name to extract data from, and format of the field seperated by a ":". '
+            'Either regular expression with named capture groups or a '
+            'similar syntax to the output template can also be used. '
+            'The parsed parameters replace any existing values and can be use in output template'
+            'This option can be used multiple times. '
+            'Example: --parse-metadata "title:%(artist)s - %(title)s" matches a title like '
             '"Coldplay - Paradise". '
-            'Example (regex): --metadata-from-title "(?P<artist>.+?) - (?P<title>.+)"'))
+            'Example (regex): --parse-metadata "description:Artist - (?P<artist>.+?)"'))
     postproc.add_option(
         '--xattrs',
         action='store_true', dest='xattrs', default=False,
@@ -1051,15 +1117,15 @@ def parseOpts(overrideArguments=None):
     postproc.add_option(
         '--prefer-avconv', '--no-prefer-ffmpeg',
         action='store_false', dest='prefer_ffmpeg',
-        help='Prefer avconv over ffmpeg for running the postprocessors')
+        help=optparse.SUPPRESS_HELP)
     postproc.add_option(
         '--prefer-ffmpeg', '--no-prefer-avconv',
-        action='store_true', dest='prefer_ffmpeg',
-        help='Prefer ffmpeg over avconv for running the postprocessors (default)')
+        action='store_true', dest='prefer_ffmpeg', default=True,
+        help=optparse.SUPPRESS_HELP)
     postproc.add_option(
         '--ffmpeg-location', '--avconv-location', metavar='PATH',
         dest='ffmpeg_location',
-        help='Location of the ffmpeg/avconv binary; either the path to the binary or its containing directory.')
+        help='Location of the ffmpeg binary; either the path to the binary or its containing directory')
     postproc.add_option(
         '--exec',
         metavar='CMD', dest='exec_cmd',
@@ -1069,12 +1135,14 @@ def parseOpts(overrideArguments=None):
         metavar='FORMAT', dest='convertsubtitles', default=None,
         help='Convert the subtitles to other format (currently supported: srt|ass|vtt|lrc)')
 
-    sponskrub = optparse.OptionGroup(parser, 'SponSkrub Options (SponsorBlock)')
+    sponskrub = optparse.OptionGroup(parser, 'SponSkrub (SponsorBlock) Options', description=(
+        'SponSkrub (https://github.com/pukkandan/SponSkrub) is a utility to mark/remove sponsor segments '
+        'from downloaded YouTube videos using SponsorBlock API (https://sponsor.ajay.app)'))
     sponskrub.add_option(
         '--sponskrub',
         action='store_true', dest='sponskrub', default=None,
         help=(
-            'Use sponskrub to mark sponsored sections with the data available in SponsorBlock API. '
+            'Use sponskrub to mark sponsored sections. '
             'This is enabled by default if the sponskrub binary exists (Youtube only)'))
     sponskrub.add_option(
         '--no-sponskrub',
@@ -1099,7 +1167,7 @@ def parseOpts(overrideArguments=None):
     sponskrub.add_option(
         '--sponskrub-location', metavar='PATH',
         dest='sponskrub_path', default='',
-        help='Location of the sponskrub binary; either the path to the binary or its containing directory.')
+        help='Location of the sponskrub binary; either the path to the binary or its containing directory')
     sponskrub.add_option(
         '--sponskrub-args', dest='sponskrub_args', metavar='ARGS',
         help=optparse.SUPPRESS_HELP)
@@ -1108,11 +1176,11 @@ def parseOpts(overrideArguments=None):
     extractor.add_option(
         '--allow-dynamic-mpd', '--no-ignore-dynamic-mpd',
         action='store_true', dest='dynamic_mpd', default=True,
-        help='Process dynamic DASH manifests (default)')
+        help='Process dynamic DASH manifests (default) (Alias: --no-ignore-dynamic-mpd)')
     extractor.add_option(
         '--ignore-dynamic-mpd', '--no-allow-dynamic-mpd',
         action='store_false', dest='dynamic_mpd',
-        help='Do not process dynamic DASH manifests')
+        help='Do not process dynamic DASH manifests (Alias: --no-allow-dynamic-mpd)')
 
     parser.add_option_group(general)
     parser.add_option_group(network)
@@ -1142,33 +1210,70 @@ def parseOpts(overrideArguments=None):
                 return [a.decode(preferredencoding(), 'replace') for a in conf]
             return conf
 
-        command_line_conf = compat_conf(sys.argv[1:])
-        opts, args = parser.parse_args(command_line_conf)
+        configs = {
+            'command-line': compat_conf(sys.argv[1:]),
+            'custom': [], 'home': [], 'portable': [], 'user': [], 'system': []}
+        paths = {'command-line': False}
+        opts, args = parser.parse_args(configs['command-line'])
 
-        system_conf = user_conf = custom_conf = []
+        def get_configs():
+            if '--config-location' in configs['command-line']:
+                location = compat_expanduser(opts.config_location)
+                if os.path.isdir(location):
+                    location = os.path.join(location, 'youtube-dlc.conf')
+                if not os.path.exists(location):
+                    parser.error('config-location %s does not exist.' % location)
+                configs['custom'] = _readOptions(location, default=None)
+                if configs['custom'] is None:
+                    configs['custom'] = []
+                else:
+                    paths['custom'] = location
+            if '--ignore-config' in configs['command-line']:
+                return
+            if '--ignore-config' in configs['custom']:
+                return
 
-        if '--config-location' in command_line_conf:
-            location = compat_expanduser(opts.config_location)
-            if os.path.isdir(location):
-                location = os.path.join(location, 'youtube-dlc.conf')
-            if not os.path.exists(location):
-                parser.error('config-location %s does not exist.' % location)
-            custom_conf = _readOptions(location)
-        elif '--ignore-config' in command_line_conf:
-            pass
-        else:
-            system_conf = _readOptions('/etc/youtube-dlc.conf')
-            if '--ignore-config' not in system_conf:
-                user_conf = _readUserConf()
+            def read_options(path, user=False):
+                for package in ('yt-dlp', 'youtube-dlc'):
+                    if user:
+                        config, current_path = _readUserConf(package, default=None)
+                    else:
+                        current_path = os.path.join(path, '%s.conf' % package)
+                        config = _readOptions(current_path, default=None)
+                    if config is not None:
+                        return config, current_path
+                return [], None
 
-        argv = system_conf + user_conf + custom_conf + command_line_conf
+            configs['portable'], paths['portable'] = read_options(get_executable_path())
+            if '--ignore-config' in configs['portable']:
+                return
+
+            def get_home_path():
+                opts = parser.parse_args(configs['portable'] + configs['custom'] + configs['command-line'])[0]
+                return expand_path(opts.paths.get('home', '')).strip()
+
+            configs['home'], paths['home'] = read_options(get_home_path())
+            if '--ignore-config' in configs['home']:
+                return
+
+            configs['system'], paths['system'] = read_options('/etc')
+            if '--ignore-config' in configs['system']:
+                return
+
+            configs['user'], paths['user'] = read_options('', True)
+            if '--ignore-config' in configs['user']:
+                configs['system'], paths['system'] = [], None
+
+        get_configs()
+        argv = configs['system'] + configs['user'] + configs['home'] + configs['portable'] + configs['custom'] + configs['command-line']
         opts, args = parser.parse_args(argv)
         if opts.verbose:
-            for conf_label, conf in (
-                    ('System config', system_conf),
-                    ('User config', user_conf),
-                    ('Custom config', custom_conf),
-                    ('Command-line args', command_line_conf)):
-                write_string('[debug] %s: %s\n' % (conf_label, repr(_hide_login_info(conf))))
+            for label in ('System', 'User', 'Portable', 'Home', 'Custom', 'Command-line'):
+                key = label.lower()
+                if paths.get(key) is None:
+                    continue
+                if paths[key]:
+                    write_string('[debug] %s config file: %s\n' % (label, paths[key]))
+                write_string('[debug] %s config: %s\n' % (label, repr(_hide_login_info(configs[key]))))
 
     return parser, opts, args

youtube_dlc/postprocessor/__init__.py

@@ -16,7 +16,9 @@ from .ffmpeg import (
 )
 from .xattrpp import XAttrMetadataPP
 from .execafterdownload import ExecAfterDownloadPP
-from .metadatafromtitle import MetadataFromTitlePP
+from .metadatafromfield import MetadataFromFieldPP
+from .metadatafromfield import MetadataFromTitlePP
+from .movefilesafterdownload import MoveFilesAfterDownloadPP
 from .sponskrub import SponSkrubPP
 
 
@@ -38,7 +40,9 @@ __all__ = [
     'FFmpegSubtitlesConvertorPP',
     'FFmpegVideoConvertorPP',
     'FFmpegVideoRemuxerPP',
+    'MetadataFromFieldPP',
     'MetadataFromTitlePP',
+    'MoveFilesAfterDownloadPP',
     'SponSkrubPP',
     'XAttrMetadataPP',
 ]

youtube_dlc/postprocessor/common.py

@@ -2,10 +2,11 @@ from __future__ import unicode_literals
 
 import os
 
+from ..compat import compat_str
 from ..utils import (
-    PostProcessingError,
     cli_configuration_args,
     encodeFilename,
+    PostProcessingError,
 )
 
 
@@ -33,12 +34,17 @@ class PostProcessor(object):
 
     def __init__(self, downloader=None):
         self._downloader = downloader
-        if not hasattr(self, 'PP_NAME'):
-            self.PP_NAME = self.__class__.__name__[:-2]
+        self.PP_NAME = self.pp_key()
 
-    def to_screen(self, text, *args, **kwargs):
+    @classmethod
+    def pp_key(cls):
+        name = cls.__name__[:-2]
+        return compat_str(name[6:]) if name[:6].lower() == 'ffmpeg' else name
+
+    def to_screen(self, text, prefix=True, *args, **kwargs):
+        tag = '[%s] ' % self.PP_NAME if prefix else ''
         if self._downloader:
-            return self._downloader.to_screen('[%s] %s' % (self.PP_NAME, text), *args, **kwargs)
+            return self._downloader.to_screen('%s%s' % (tag, text), *args, **kwargs)
 
     def report_warning(self, text, *args, **kwargs):
         if self._downloader:
@@ -48,9 +54,10 @@ class PostProcessor(object):
         if self._downloader:
             return self._downloader.report_error(text, *args, **kwargs)
 
-    def write_debug(self, text, *args, **kwargs):
-        if self.get_param('verbose', False):
-            return self._downloader.to_screen('[debug] %s' % text, *args, **kwargs)
+    def write_debug(self, text, prefix=True, *args, **kwargs):
+        tag = '[debug] ' if prefix else ''
+        if self.get_param('verbose', False) and self._downloader:
+            return self._downloader.to_screen('%s%s' % (tag, text), *args, **kwargs)
 
     def get_param(self, name, default=None, *args, **kwargs):
         if self._downloader:
@@ -84,11 +91,11 @@ class PostProcessor(object):
         except Exception:
             self.report_warning(errnote)
 
-    def _configuration_args(self, default=[]):
-        args = self.get_param('postprocessor_args', {})
-        if isinstance(args, list):  # for backward compatibility
-            args = {'default': args, 'sponskrub': []}
-        return cli_configuration_args(args, self.PP_NAME.lower(), args.get('default', []))
+    def _configuration_args(self, default=[], exe=None):
+        key = self.pp_key().lower()
+        args, is_compat = cli_configuration_args(
+            self._downloader.params, 'postprocessor_args', key, default, exe)
+        return args if not is_compat or key != 'sponskrub' else default
 
 
 class AudioConversionError(PostProcessingError):

youtube_dlc/postprocessor/embedthumbnail.py

@@ -4,6 +4,15 @@ from __future__ import unicode_literals
 
 import os
 import subprocess
+import struct
+import re
+import base64
+
+try:
+    import mutagen
+    _has_mutagen = True
+except ImportError:
+    _has_mutagen = False
 
 from .ffmpeg import FFmpegPostProcessor
 
@@ -11,11 +20,12 @@ from ..utils import (
     check_executable,
     encodeArgument,
     encodeFilename,
+    error_to_compat_str,
     PostProcessingError,
     prepend_extension,
+    process_communicate_or_kill,
     replace_extension,
     shell_quote,
-    process_communicate_or_kill,
 )
 
 
@@ -24,7 +34,6 @@ class EmbedThumbnailPPError(PostProcessingError):
 
 
 class EmbedThumbnailPP(FFmpegPostProcessor):
-    PP_NAME = 'EmbedThumbnail'
 
     def __init__(self, downloader=None, already_have_thumbnail=False):
         super(EmbedThumbnailPP, self).__init__(downloader)
@@ -74,75 +83,103 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             # Rename back to unescaped for further processing
             os.rename(encodeFilename(escaped_thumbnail_jpg_filename), encodeFilename(thumbnail_jpg_filename))
             thumbnail_filename = thumbnail_jpg_filename
+            thumbnail_ext = 'jpg'
 
+        success = True
         if info['ext'] == 'mp3':
             options = [
                 '-c', 'copy', '-map', '0:0', '-map', '1:0', '-id3v2_version', '3',
                 '-metadata:s:v', 'title="Album cover"', '-metadata:s:v', 'comment="Cover (front)"']
 
             self.to_screen('Adding thumbnail to "%s"' % filename)
-
             self.run_ffmpeg_multiple_files([filename, thumbnail_filename], temp_filename, options)
 
-            if not self._already_have_thumbnail:
-                os.remove(encodeFilename(thumbnail_filename))
-            os.remove(encodeFilename(filename))
-            os.rename(encodeFilename(temp_filename), encodeFilename(filename))
+        elif info['ext'] in ['mkv', 'mka']:
+            options = ['-c', 'copy', '-map', '0', '-dn']
 
-        elif info['ext'] == 'mkv':
-            old_thumbnail_filename = thumbnail_filename
-            thumbnail_filename = os.path.join(os.path.dirname(old_thumbnail_filename), 'cover.jpg')
-            if os.path.exists(thumbnail_filename):
-                os.remove(encodeFilename(thumbnail_filename))
-            os.rename(encodeFilename(old_thumbnail_filename), encodeFilename(thumbnail_filename))
-
-            options = [
-                '-c', 'copy', '-map', '0', '-dn',
-                '-attach', thumbnail_filename, '-metadata:s:t', 'mimetype=image/jpeg']
+            mimetype = 'image/%s' % ('png' if thumbnail_ext == 'png' else 'jpeg')
+            old_stream, new_stream = self.get_stream_number(
+                filename, ('tags', 'mimetype'), mimetype)
+            if old_stream is not None:
+                options.extend(['-map', '-0:%d' % old_stream])
+                new_stream -= 1
+            options.extend([
+                '-attach', thumbnail_filename,
+                '-metadata:s:%d' % new_stream, 'mimetype=%s' % mimetype,
+                '-metadata:s:%d' % new_stream, 'filename=cover.%s' % thumbnail_ext])
 
             self.to_screen('Adding thumbnail to "%s"' % filename)
+            self.run_ffmpeg(filename, temp_filename, options)
 
-            self.run_ffmpeg_multiple_files([filename], temp_filename, options)
+        elif info['ext'] in ['m4a', 'mp4', 'mov']:
+            try:
+                options = ['-c', 'copy', '-map', '0', '-dn', '-map', '1']
 
-            if not self._already_have_thumbnail:
-                os.remove(encodeFilename(thumbnail_filename))
-            else:
-                os.rename(encodeFilename(thumbnail_filename), encodeFilename(old_thumbnail_filename))
-            os.remove(encodeFilename(filename))
-            os.rename(encodeFilename(temp_filename), encodeFilename(filename))
+                old_stream, new_stream = self.get_stream_number(
+                    filename, ('disposition', 'attached_pic'), 1)
+                if old_stream is not None:
+                    options.extend(['-map', '-0:%d' % old_stream])
+                    new_stream -= 1
+                options.extend(['-disposition:%s' % new_stream, 'attached_pic'])
 
-        elif info['ext'] in ['m4a', 'mp4']:
-            if not check_executable('AtomicParsley', ['-v']):
-                raise EmbedThumbnailPPError('AtomicParsley was not found. Please install.')
+                self.to_screen('Adding thumbnail to "%s"' % filename)
+                self.run_ffmpeg_multiple_files([filename, thumbnail_filename], temp_filename, options)
 
-            cmd = [encodeFilename('AtomicParsley', True),
-                   encodeFilename(filename, True),
-                   encodeArgument('--artwork'),
-                   encodeFilename(thumbnail_filename, True),
-                   encodeArgument('-o'),
-                   encodeFilename(temp_filename, True)]
+            except PostProcessingError as err:
+                self.report_warning('unable to embed using ffprobe & ffmpeg; %s' % error_to_compat_str(err))
+                if not check_executable('AtomicParsley', ['-v']):
+                    raise EmbedThumbnailPPError('AtomicParsley was not found. Please install.')
 
-            self.to_screen('Adding thumbnail to "%s"' % filename)
+                cmd = [encodeFilename('AtomicParsley', True),
+                       encodeFilename(filename, True),
+                       encodeArgument('--artwork'),
+                       encodeFilename(thumbnail_filename, True),
+                       encodeArgument('-o'),
+                       encodeFilename(temp_filename, True)]
+                cmd += [encodeArgument(o) for o in self._configuration_args(exe='AtomicParsley')]
 
-            self.verbose_message('AtomicParsley command line: %s' % shell_quote(cmd))
+                self.to_screen('Adding thumbnail to "%s"' % filename)
+                self.write_debug('AtomicParsley command line: %s' % shell_quote(cmd))
+                p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                stdout, stderr = process_communicate_or_kill(p)
+                if p.returncode != 0:
+                    msg = stderr.decode('utf-8', 'replace').strip()
+                    raise EmbedThumbnailPPError(msg)
+                # for formats that don't support thumbnails (like 3gp) AtomicParsley
+                # won't create to the temporary file
+                if b'No changes' in stdout:
+                    self.report_warning('The file format doesn\'t support embedding a thumbnail')
+                    success = False
 
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-            stdout, stderr = process_communicate_or_kill(p)
+        elif info['ext'] in ['ogg', 'opus']:
+            if not _has_mutagen:
+                raise EmbedThumbnailPPError('module mutagen was not found. Please install using `python -m pip install mutagen`')
+            size_regex = r',\s*(?P<w>\d+)x(?P<h>\d+)\s*[,\[]'
+            size_result = self.run_ffmpeg(thumbnail_filename, thumbnail_filename, ['-hide_banner'])
+            mobj = re.search(size_regex, size_result)
+            width, height = int(mobj.group('w')), int(mobj.group('h'))
+            mimetype = ('image/%s' % ('png' if thumbnail_ext == 'png' else 'jpeg')).encode('ascii')
 
-            if p.returncode != 0:
-                msg = stderr.decode('utf-8', 'replace').strip()
-                raise EmbedThumbnailPPError(msg)
+            # https://xiph.org/flac/format.html#metadata_block_picture
+            data = bytearray()
+            data += struct.pack('>II', 3, len(mimetype))
+            data += mimetype
+            data += struct.pack('>IIIIII', 0, width, height, 8, 0, os.stat(thumbnail_filename).st_size)  # 32 if png else 24
+
+            fin = open(thumbnail_filename, "rb")
+            data += fin.read()
+            fin.close()
+
+            temp_filename = filename
+            f = mutagen.File(temp_filename)
+            f.tags['METADATA_BLOCK_PICTURE'] = base64.b64encode(data).decode('ascii')
+            f.save()
 
-            if not self._already_have_thumbnail:
-                os.remove(encodeFilename(thumbnail_filename))
-            # for formats that don't support thumbnails (like 3gp) AtomicParsley
-            # won't create to the temporary file
-            if b'No changes' in stdout:
-                self.report_warning('The file format doesn\'t support embedding a thumbnail')
-            else:
-                os.remove(encodeFilename(filename))
-                os.rename(encodeFilename(temp_filename), encodeFilename(filename))
         else:
-            raise EmbedThumbnailPPError('Only mp3, mkv, m4a and mp4 are supported for thumbnail embedding for now.')
+            raise EmbedThumbnailPPError('Supported filetypes for thumbnail embedding are: mp3, mkv/mka, ogg/opus, m4a/mp4/mov')
 
-        return [], info
+        if success and temp_filename != filename:
+            os.remove(encodeFilename(filename))
+            os.rename(encodeFilename(temp_filename), encodeFilename(filename))
+        files_to_delete = [] if self._already_have_thumbnail else [thumbnail_filename]
+        return files_to_delete, info

youtube_dlc/postprocessor/execafterdownload.py

@@ -11,12 +11,15 @@ from ..utils import (
 
 
 class ExecAfterDownloadPP(PostProcessor):
-    PP_NAME = 'Exec'
 
     def __init__(self, downloader, exec_cmd):
         super(ExecAfterDownloadPP, self).__init__(downloader)
         self.exec_cmd = exec_cmd
 
+    @classmethod
+    def pp_key(cls):
+        return 'Exec'
+
     def run(self, information):
         cmd = self.exec_cmd
         if '{}' not in cmd:

youtube_dlc/postprocessor/ffmpeg.py

@@ -5,6 +5,7 @@ import os
 import subprocess
 import time
 import re
+import json
 
 
 from .common import AudioConversionError, PostProcessor
@@ -20,8 +21,9 @@ from ..utils import (
     subtitles_filename,
     dfxp2srt,
     ISO639Utils,
-    replace_extension,
     process_communicate_or_kill,
+    replace_extension,
+    traverse_dict,
 )
 
 
@@ -54,14 +56,12 @@ class FFmpegPostProcessorError(PostProcessingError):
 
 class FFmpegPostProcessor(PostProcessor):
     def __init__(self, downloader=None):
-        if not hasattr(self, 'PP_NAME'):
-            self.PP_NAME = self.__class__.__name__[6:-2]  # Remove ffmpeg from the front
         PostProcessor.__init__(self, downloader)
         self._determine_executables()
 
     def check_version(self):
         if not self.available:
-            raise FFmpegPostProcessorError('ffmpeg or avconv not found. Please install one.')
+            raise FFmpegPostProcessorError('ffmpeg not found. Please install one.')
 
         required_version = '10-0' if self.basename == 'avconv' else '1.0'
         if is_outdated_version(
@@ -104,7 +104,7 @@ class FFmpegPostProcessor(PostProcessor):
                 if not os.path.exists(location):
                     self.report_warning(
                         'ffmpeg-location %s does not exist! '
-                        'Continuing without avconv/ffmpeg.' % (location))
+                        'Continuing without ffmpeg.' % (location))
                     self._versions = {}
                     return
                 elif not os.path.isdir(location):
@@ -112,7 +112,7 @@ class FFmpegPostProcessor(PostProcessor):
                     if basename not in programs:
                         self.report_warning(
                             'Cannot identify executable %s, its basename should be one of %s. '
-                            'Continuing without avconv/ffmpeg.' %
+                            'Continuing without ffmpeg.' %
                             (location, ', '.join(programs)))
                         self._versions = {}
                         return None
@@ -165,7 +165,7 @@ class FFmpegPostProcessor(PostProcessor):
 
     def get_audio_codec(self, path):
         if not self.probe_available and not self.available:
-            raise PostProcessingError('ffprobe/avprobe and ffmpeg/avconv not found. Please install one.')
+            raise PostProcessingError('ffprobe and ffmpeg not found. Please install one.')
         try:
             if self.probe_available:
                 cmd = [
@@ -203,13 +203,44 @@ class FFmpegPostProcessor(PostProcessor):
                 return mobj.group(1)
         return None
 
+    def get_metadata_object(self, path, opts=[]):
+        if self.probe_basename != 'ffprobe':
+            if self.probe_available:
+                self.report_warning('Only ffprobe is supported for metadata extraction')
+            raise PostProcessingError('ffprobe not found. Please install.')
+        self.check_version()
+
+        cmd = [
+            encodeFilename(self.probe_executable, True),
+            encodeArgument('-hide_banner'),
+            encodeArgument('-show_format'),
+            encodeArgument('-show_streams'),
+            encodeArgument('-print_format'),
+            encodeArgument('json'),
+        ]
+
+        cmd += opts
+        cmd.append(encodeFilename(self._ffmpeg_filename_argument(path), True))
+        if self._downloader.params.get('verbose', False):
+            self._downloader.to_screen('[debug] ffprobe command line: %s' % shell_quote(cmd))
+        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
+        stdout, stderr = p.communicate()
+        return json.loads(stdout.decode('utf-8', 'replace'))
+
+    def get_stream_number(self, path, keys, value):
+        streams = self.get_metadata_object(path)['streams']
+        num = next(
+            (i for i, stream in enumerate(streams) if traverse_dict(stream, keys, casesense=False) == value),
+            None)
+        return num, len(streams)
+
     def run_ffmpeg_multiple_files(self, input_paths, out_path, opts):
         self.check_version()
 
         oldest_mtime = min(
             os.stat(encodeFilename(path)).st_mtime for path in input_paths)
 
-        opts += self._configuration_args()
+        opts += self._configuration_args(exe=self.basename)
 
         files_cmd = []
         for path in input_paths:
@@ -229,19 +260,23 @@ class FFmpegPostProcessor(PostProcessor):
         p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
         stdout, stderr = process_communicate_or_kill(p)
         if p.returncode != 0:
-            stderr = stderr.decode('utf-8', 'replace')
-            msg = stderr.strip().split('\n')[-1]
-            raise FFmpegPostProcessorError(msg)
+            stderr = stderr.decode('utf-8', 'replace').strip()
+            if self._downloader.params.get('verbose', False):
+                self.report_error(stderr)
+            raise FFmpegPostProcessorError(stderr.split('\n')[-1])
         self.try_utime(out_path, oldest_mtime, oldest_mtime)
+        return stderr.decode('utf-8', 'replace')
 
     def run_ffmpeg(self, path, out_path, opts):
-        self.run_ffmpeg_multiple_files([path], out_path, opts)
+        return self.run_ffmpeg_multiple_files([path], out_path, opts)
 
     def _ffmpeg_filename_argument(self, fn):
         # Always use 'file:' because the filename may contain ':' (ffmpeg
         # interprets that as a protocol) or can start with '-' (-- is broken in
         # ffmpeg, see https://ffmpeg.org/trac/ffmpeg/ticket/2127 for details)
         # Also leave '-' intact in order not to break streaming to stdout.
+        if fn.startswith(('http://', 'https://')):
+            return fn
         return 'file:' + fn if fn != '-' else fn
 
 
@@ -351,21 +386,35 @@ class FFmpegExtractAudioPP(FFmpegPostProcessor):
 class FFmpegVideoRemuxerPP(FFmpegPostProcessor):
     def __init__(self, downloader=None, preferedformat=None):
         super(FFmpegVideoRemuxerPP, self).__init__(downloader)
-        self._preferedformat = preferedformat
+        self._preferedformats = preferedformat.lower().split('/')
 
     def run(self, information):
         path = information['filepath']
-        if information['ext'] == self._preferedformat:
-            self.to_screen('Not remuxing video file %s - already is in target format %s' % (path, self._preferedformat))
+        sourceext, targetext = information['ext'].lower(), None
+        for pair in self._preferedformats:
+            kv = pair.split('>')
+            if len(kv) == 1 or kv[0].strip() == sourceext:
+                targetext = kv[-1].strip()
+                break
+
+        _skip_msg = (
+            'could not find a mapping for %s' if not targetext
+            else 'already is in target format %s' if sourceext == targetext
+            else None)
+        if _skip_msg:
+            self.to_screen('Not remuxing media file %s; %s' % (path, _skip_msg % sourceext))
             return [], information
+
         options = ['-c', 'copy', '-map', '0', '-dn']
-        prefix, sep, ext = path.rpartition('.')
-        outpath = prefix + sep + self._preferedformat
-        self.to_screen('Remuxing video from %s to %s, Destination: ' % (information['ext'], self._preferedformat) + outpath)
+        if targetext in ['mp4', 'm4a', 'mov']:
+            options.extend(['-movflags', '+faststart'])
+        prefix, sep, oldext = path.rpartition('.')
+        outpath = prefix + sep + targetext
+        self.to_screen('Remuxing video from %s to %s; Destination: %s' % (sourceext, targetext, outpath))
         self.run_ffmpeg(path, outpath, options)
         information['filepath'] = outpath
-        information['format'] = self._preferedformat
-        information['ext'] = self._preferedformat
+        information['format'] = targetext
+        information['ext'] = targetext
         return [path], information
 
 
@@ -408,18 +457,22 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
         sub_langs = []
         sub_filenames = []
         webm_vtt_warn = False
+        mp4_ass_warn = False
 
         for lang, sub_info in subtitles.items():
             sub_ext = sub_info['ext']
             if sub_ext == 'json':
-                self.to_screen('JSON subtitles cannot be embedded')
+                self.report_warning('JSON subtitles cannot be embedded')
             elif ext != 'webm' or ext == 'webm' and sub_ext == 'vtt':
                 sub_langs.append(lang)
                 sub_filenames.append(subtitles_filename(filename, lang, sub_ext, ext))
             else:
                 if not webm_vtt_warn and ext == 'webm' and sub_ext != 'vtt':
                     webm_vtt_warn = True
-                    self.to_screen('Only WebVTT subtitles can be embedded in webm files')
+                    self.report_warning('Only WebVTT subtitles can be embedded in webm files')
+            if not mp4_ass_warn and ext == 'mp4' and sub_ext == 'ass':
+                mp4_ass_warn = True
+                self.report_warning('ASS subtitles cannot be properly embedded in mp4 files; expect issues')
 
         if not sub_langs:
             return [], information
@@ -443,7 +496,7 @@ class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
             opts.extend(['-metadata:s:s:%d' % i, 'language=%s' % lang_code])
 
         temp_filename = prepend_extension(filename, 'temp')
-        self.to_screen('Embedding subtitles in \'%s\'' % filename)
+        self.to_screen('Embedding subtitles in "%s"' % filename)
         self.run_ffmpeg_multiple_files(input_files, temp_filename, opts)
         os.remove(encodeFilename(filename))
         os.rename(encodeFilename(temp_filename), encodeFilename(filename))
@@ -473,7 +526,6 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
         # 1. https://kdenlive.org/en/project/adding-meta-data-to-mp4-video/
         # 2. https://wiki.multimedia.cx/index.php/FFmpeg_Metadata
         # 3. https://kodi.wiki/view/Video_file_tagging
-        # 4. http://atomicparsley.sourceforge.net/mpeg-4files.html
 
         add('title', ('track', 'title'))
         add('date', 'upload_date')
@@ -526,6 +578,18 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
                 in_filenames.append(metadata_filename)
                 options.extend(['-map_metadata', '1'])
 
+        if '__infojson_filepath' in info and info['ext'] in ('mkv', 'mka'):
+            old_stream, new_stream = self.get_stream_number(
+                filename, ('tags', 'mimetype'), 'application/json')
+            if old_stream is not None:
+                options.extend(['-map', '-0:%d' % old_stream])
+                new_stream -= 1
+
+            options.extend([
+                '-attach', info['__infojson_filepath'],
+                '-metadata:s:%d' % new_stream, 'mimetype=application/json'
+            ])
+
         self.to_screen('Adding metadata to \'%s\'' % filename)
         self.run_ffmpeg_multiple_files(in_filenames, temp_filename, options)
         if chapters:

youtube_dlc/postprocessor/metadatafromfield.py

@@ -0,0 +1,66 @@
+from __future__ import unicode_literals
+
+import re
+
+from .common import PostProcessor
+from ..compat import compat_str
+
+
+class MetadataFromFieldPP(PostProcessor):
+    regex = r'(?P<field>\w+):(?P<format>.+)$'
+
+    def __init__(self, downloader, formats):
+        PostProcessor.__init__(self, downloader)
+        assert isinstance(formats, (list, tuple))
+        self._data = []
+        for f in formats:
+            assert isinstance(f, compat_str)
+            match = re.match(self.regex, f)
+            assert match is not None
+            self._data.append({
+                'field': match.group('field'),
+                'format': match.group('format'),
+                'regex': self.format_to_regex(match.group('format'))})
+
+    def format_to_regex(self, fmt):
+        r"""
+        Converts a string like
+           '%(title)s - %(artist)s'
+        to a regex like
+           '(?P<title>.+)\ \-\ (?P<artist>.+)'
+        """
+        if not re.search(r'%\(\w+\)s', fmt):
+            return fmt
+        lastpos = 0
+        regex = ''
+        # replace %(..)s with regex group and escape other string parts
+        for match in re.finditer(r'%\((\w+)\)s', fmt):
+            regex += re.escape(fmt[lastpos:match.start()])
+            regex += r'(?P<' + match.group(1) + r'>[^\r\n]+)'
+            lastpos = match.end()
+        if lastpos < len(fmt):
+            regex += re.escape(fmt[lastpos:])
+        return regex
+
+    def run(self, info):
+        for dictn in self._data:
+            field, regex = dictn['field'], dictn['regex']
+            if field not in info:
+                self.report_warning('Video doesnot have a %s' % field)
+                continue
+            self.write_debug('Searching for r"%s" in %s' % (regex, field))
+            match = re.search(regex, info[field])
+            if match is None:
+                self.report_warning('Could not interpret video %s as "%s"' % (field, dictn['format']))
+                continue
+            for attribute, value in match.groupdict().items():
+                info[attribute] = value
+                self.to_screen('parsed %s from %s: %s' % (attribute, field, value if value is not None else 'NA'))
+        return [], info
+
+
+class MetadataFromTitlePP(MetadataFromFieldPP):  # for backward compatibility
+    def __init__(self, downloader, titleformat):
+        super(MetadataFromTitlePP, self).__init__(downloader, ['title:%s' % titleformat])
+        self._titleformat = titleformat
+        self._titleregex = self._data[0]['regex']

youtube_dlc/postprocessor/metadatafromtitle.py

@@ -1,44 +0,0 @@
-from __future__ import unicode_literals
-
-import re
-
-from .common import PostProcessor
-
-
-class MetadataFromTitlePP(PostProcessor):
-    def __init__(self, downloader, titleformat):
-        super(MetadataFromTitlePP, self).__init__(downloader)
-        self._titleformat = titleformat
-        self._titleregex = (self.format_to_regex(titleformat)
-                            if re.search(r'%\(\w+\)s', titleformat)
-                            else titleformat)
-
-    def format_to_regex(self, fmt):
-        r"""
-        Converts a string like
-           '%(title)s - %(artist)s'
-        to a regex like
-           '(?P<title>.+)\ \-\ (?P<artist>.+)'
-        """
-        lastpos = 0
-        regex = ''
-        # replace %(..)s with regex group and escape other string parts
-        for match in re.finditer(r'%\((\w+)\)s', fmt):
-            regex += re.escape(fmt[lastpos:match.start()])
-            regex += r'(?P<' + match.group(1) + '>.+)'
-            lastpos = match.end()
-        if lastpos < len(fmt):
-            regex += re.escape(fmt[lastpos:])
-        return regex
-
-    def run(self, info):
-        title = info['title']
-        match = re.match(self._titleregex, title)
-        if match is None:
-            self.to_screen('Could not interpret title of video as "%s"' % self._titleformat)
-            return [], info
-        for attribute, value in match.groupdict().items():
-            info[attribute] = value
-            self.to_screen('parsed %s: %s' % (attribute, value if value is not None else 'NA'))
-
-        return [], info

youtube_dlc/postprocessor/movefilesafterdownload.py

@@ -0,0 +1,53 @@
+from __future__ import unicode_literals
+import os
+import shutil
+
+from .common import PostProcessor
+from ..utils import (
+    encodeFilename,
+    make_dir,
+    PostProcessingError,
+)
+from ..compat import compat_str
+
+
+class MoveFilesAfterDownloadPP(PostProcessor):
+
+    def __init__(self, downloader, files_to_move):
+        PostProcessor.__init__(self, downloader)
+        self.files_to_move = files_to_move
+
+    @classmethod
+    def pp_key(cls):
+        return 'MoveFiles'
+
+    def run(self, info):
+        dl_path, dl_name = os.path.split(encodeFilename(info['filepath']))
+        finaldir = info.get('__finaldir', dl_path)
+        finalpath = os.path.join(finaldir, dl_name)
+        self.files_to_move[info['filepath']] = finalpath
+
+        for oldfile, newfile in self.files_to_move.items():
+            if not newfile:
+                newfile = os.path.join(finaldir, os.path.basename(encodeFilename(oldfile)))
+            oldfile, newfile = compat_str(oldfile), compat_str(newfile)
+            if os.path.abspath(encodeFilename(oldfile)) == os.path.abspath(encodeFilename(newfile)):
+                continue
+            if not os.path.exists(encodeFilename(oldfile)):
+                self.report_warning('File "%s" cannot be found' % oldfile)
+                continue
+            if os.path.exists(encodeFilename(newfile)):
+                if self.get_param('overwrites', True):
+                    self.report_warning('Replacing existing file "%s"' % newfile)
+                    os.path.remove(encodeFilename(newfile))
+                else:
+                    self.report_warning(
+                        'Cannot move file "%s" out of temporary directory since "%s" already exists. '
+                        % (oldfile, newfile))
+                    continue
+            make_dir(newfile, PostProcessingError)
+            self.to_screen('Moving file "%s" to "%s"' % (oldfile, newfile))
+            shutil.move(oldfile, newfile)  # os.rename cannot move between volumes
+
+        info['filepath'] = compat_str(finalpath)
+        return [], info

youtube_dlc/postprocessor/sponskrub.py

@@ -7,22 +7,23 @@ from ..compat import compat_shlex_split
 from ..utils import (
     check_executable,
     encodeArgument,
+    encodeFilename,
     shell_quote,
+    str_or_none,
     PostProcessingError,
+    prepend_extension,
 )
 
 
 class SponSkrubPP(PostProcessor):
     _temp_ext = 'spons'
-    _def_args = []
     _exe_name = 'sponskrub'
 
     def __init__(self, downloader, path='', args=None, ignoreerror=False, cut=False, force=False):
         PostProcessor.__init__(self, downloader)
         self.force = force
         self.cutout = cut
-        self.args = ['-chapter'] if not cut else []
-        self.args += self._configuration_args(self._def_args) if args is None else compat_shlex_split(args)
+        self.args = str_or_none(args) or ''  # For backward compatibility
         self.path = self.get_exe(path)
 
         if not ignoreerror and self.path is None:
@@ -58,13 +59,15 @@ class SponSkrubPP(PostProcessor):
                 self.report_warning('If sponskrub is run multiple times, unintended parts of the video could be cut out.')
 
         filename = information['filepath']
-        temp_filename = filename + '.' + self._temp_ext + os.path.splitext(filename)[1]
-        if os.path.exists(temp_filename):
-            os.remove(temp_filename)
+        temp_filename = prepend_extension(filename, self._temp_ext)
+        if os.path.exists(encodeFilename(temp_filename)):
+            os.remove(encodeFilename(temp_filename))
 
         cmd = [self.path]
-        if self.args:
-            cmd += self.args
+        if not self.cutout:
+            cmd += ['-chapter']
+        cmd += compat_shlex_split(self.args)  # For backward compatibility
+        cmd += self._configuration_args(exe=self._exe_name)
         cmd += ['--', information['id'], filename, temp_filename]
         cmd = [encodeArgument(i) for i in cmd]
 
@@ -73,13 +76,15 @@ class SponSkrubPP(PostProcessor):
         stdout, stderr = p.communicate()
 
         if p.returncode == 0:
-            os.remove(filename)
-            os.rename(temp_filename, filename)
+            os.remove(encodeFilename(filename))
+            os.rename(encodeFilename(temp_filename), encodeFilename(filename))
             self.to_screen('Sponsor sections have been %s' % ('removed' if self.cutout else 'marked'))
         elif p.returncode == 3:
             self.to_screen('No segments in the SponsorBlock database')
         else:
-            stderr = stderr.decode('utf-8', 'replace')
-            msg = stderr.strip().split('\n')[-1]
-            raise PostProcessingError(msg if msg else 'sponskrub failed with error code %s!' % p.returncode)
+            msg = stderr.decode('utf-8', 'replace').strip() or stdout.decode('utf-8', 'replace').strip()
+            self.write_debug(msg, prefix=False)
+            line = 0 if msg[:12].lower() == 'unrecognised' else -1
+            msg = msg.split('\n')[line]
+            raise PostProcessingError(msg if msg else 'sponskrub failed with error code %s' % p.returncode)
         return [], information

youtube_dlc/update.py

@@ -32,7 +32,7 @@ def rsa_verify(message, signature, key):
 def update_self(to_screen, verbose, opener):
     """Update the program file with the latest version from the repository"""
 
-    return to_screen('Update is currently broken.\nVisit  https://github.com/pukkandan/yt-dlc/releases/latest  to get the latest version')
+    return to_screen('Update is currently broken.\nVisit  https://github.com/pukkandan/yt-dlp/releases/latest  to get the latest version')
 
     UPDATE_URL = 'https://blackjack4494.github.io//update/'
     VERSION_URL = UPDATE_URL + 'LATEST_VERSION'

youtube_dlc/utils.py

@@ -16,6 +16,7 @@ import email.header
 import errno
 import functools
 import gzip
+import imp
 import io
 import itertools
 import json
@@ -2332,8 +2333,8 @@ def bug_reports_message():
     if ytdl_is_updateable():
         update_cmd = 'type  youtube-dlc -U  to update'
     else:
-        update_cmd = 'see  https://github.com/pukkandan/yt-dlc  on how to update'
-    msg = '; please report this issue on https://github.com/pukkandan/yt-dlc .'
+        update_cmd = 'see  https://github.com/pukkandan/yt-dlp  on how to update'
+    msg = '; please report this issue on https://github.com/pukkandan/yt-dlp .'
     msg += ' Make sure you are using the latest version; %s.' % update_cmd
     msg += ' Be sure to call youtube-dlc with the --verbose flag and include its complete output.'
     return msg
@@ -2433,6 +2434,16 @@ class PostProcessingError(YoutubeDLError):
         self.msg = msg
 
 
+class ExistingVideoReached(YoutubeDLError):
+    """ --max-downloads limit has been reached. """
+    pass
+
+
+class RejectedVideoReached(YoutubeDLError):
+    """ --max-downloads limit has been reached. """
+    pass
+
+
 class MaxDownloadsReached(YoutubeDLError):
     """ --max-downloads limit has been reached. """
     pass
@@ -4089,7 +4100,8 @@ def strip_jsonp(code):
         r'\g<callback_data>', code)
 
 
-def js_to_json(code):
+def js_to_json(code, vars={}):
+    # vars is a dict of var, val pairs to substitute
     COMMENT_RE = r'/\*(?:(?!\*/).)*?\*/|//[^\n]*'
     SKIP_RE = r'\s*(?:{comment})?\s*'.format(comment=COMMENT_RE)
     INTEGER_TABLE = (
@@ -4118,6 +4130,9 @@ def js_to_json(code):
                     i = int(im.group(1), base)
                     return '"%d":' % i if v.endswith(':') else '%d' % i
 
+            if v in vars:
+                return vars[v]
+
         return '"%s"' % v
 
     return re.sub(r'''(?sx)
@@ -4642,12 +4657,35 @@ def cli_valueless_option(params, command_option, param, expected_value=True):
     return [command_option] if param == expected_value else []
 
 
-def cli_configuration_args(params, param, default=[]):
-    ex_args = params.get(param)
-    if ex_args is None:
-        return default
-    assert isinstance(ex_args, list)
-    return ex_args
+def cli_configuration_args(params, arg_name, key, default=[], exe=None):  # returns arg, for_compat
+    argdict = params.get(arg_name, {})
+    if isinstance(argdict, (list, tuple)):  # for backward compatibility
+        return argdict, True
+
+    if argdict is None:
+        return default, False
+    assert isinstance(argdict, dict)
+
+    assert isinstance(key, compat_str)
+    key = key.lower()
+
+    args = exe_args = None
+    if exe is not None:
+        assert isinstance(exe, compat_str)
+        exe = exe.lower()
+        args = argdict.get('%s+%s' % (key, exe))
+        if args is None:
+            exe_args = argdict.get(exe)
+
+    if args is None:
+        args = argdict.get(key) if key != exe else None
+    if args is None and exe_args is None:
+        args = argdict.get('default', default)
+
+    args, exe_args = args or [], exe_args or []
+    assert isinstance(args, (list, tuple))
+    assert isinstance(exe_args, (list, tuple))
+    return args + exe_args, False
 
 
 class ISO639Utils(object):
@@ -5849,3 +5887,61 @@ def clean_podcast_url(url):
                 st\.fm # https://podsights.com/docs/
             )/e
         )/''', '', url)
+
+
+_HEX_TABLE = '0123456789abcdef'
+
+
+def random_uuidv4():
+    return re.sub(r'[xy]', lambda x: _HEX_TABLE[random.randint(0, 15)], 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx')
+
+
+def make_dir(path, to_screen=None):
+    try:
+        dn = os.path.dirname(path)
+        if dn and not os.path.exists(dn):
+            os.makedirs(dn)
+        return True
+    except (OSError, IOError) as err:
+        if callable(to_screen) is not None:
+            to_screen('unable to create directory ' + error_to_compat_str(err))
+        return False
+
+
+def get_executable_path():
+    path = os.path.dirname(sys.argv[0])
+    if os.path.abspath(sys.argv[0]) != os.path.abspath(sys.executable):  # Not packaged
+        path = os.path.join(path, '..')
+    return os.path.abspath(path)
+
+
+def load_plugins(name, type, namespace):
+    plugin_info = [None]
+    classes = []
+    try:
+        plugin_info = imp.find_module(
+            name, [os.path.join(get_executable_path(), 'ytdlp_plugins')])
+        plugins = imp.load_module(name, *plugin_info)
+        for name in dir(plugins):
+            if not name.endswith(type):
+                continue
+            klass = getattr(plugins, name)
+            classes.append(klass)
+            namespace[name] = klass
+    except ImportError:
+        pass
+    finally:
+        if plugin_info[0] is not None:
+            plugin_info[0].close()
+    return classes
+
+
+def traverse_dict(dictn, keys, casesense=True):
+    if not isinstance(dictn, dict):
+        return None
+    first_key = keys[0]
+    if not casesense:
+        dictn = {key.lower(): val for key, val in dictn.items()}
+        first_key = first_key.lower()
+    value = dictn.get(first_key, None)
+    return value if len(keys) < 2 else traverse_dict(value, keys[1:], casesense)

youtube_dlc/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.01.10'
+__version__ = '2021.01.24'

ytdlp_plugins/extractor/__init__.py

@@ -0,0 +1,2 @@
+# flake8: noqa
+from .sample import SamplePluginIE

ytdlp_plugins/extractor/sample.py

@@ -0,0 +1,12 @@
+from __future__ import unicode_literals
+
+from youtube_dlc.extractor.common import InfoExtractor
+
+
+class SamplePluginIE(InfoExtractor):
+    _WORKING = False
+    IE_DESC = False
+    _VALID_URL = r'^sampleplugin:'
+
+    def _real_extract(self, url):
+        self.to_screen('URL "%s" sucessfully captured' % url)