Release 2021.01.12

Closes #15

Authored by samiksome

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.08. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.08**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version 2021.01.08
+ [debug] yt-dlp version 2021.01.10
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.08. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. youtube-dlc does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.08**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.08. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.08**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.08. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.08**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version 2021.01.08
+ [debug] yt-dlp version 2021.01.10
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is 2021.01.08. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.10. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dlc version **2021.01.08**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.10**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/6_question.md

@@ -20,8 +20,8 @@ assignees: ''
 ## Checklist
 
 <!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- Look through the README (https://github.com/blackjack4494/yt-dlc) and FAQ (https://github.com/blackjack4494/yt-dlc) for similar questions
+Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
+- Look through the README (https://github.com/pukkandan/yt-dlc) and FAQ (https://github.com/pukkandan/yt-dlc) for similar questions
 - Search the bugtracker for similar questions: https://github.com/blackjack4494/yt-dlc
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version %(version)s
+ [debug] yt-dlp version %(version)s
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -21,15 +21,15 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. youtube-dlc does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlc. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlc.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dlc version %(version)s
+ [debug] yt-dlp version %(version)s
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of youtube-dlc. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is %(version)s. If it's not, see https://github.com/pukkandan/yt-dlc on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlc. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dlc version **%(version)s**
+- [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/build.yml

@@ -161,3 +161,19 @@ jobs:
         asset_path: ./SHA2-256SUMS
         asset_name: SHA2-256SUMS
         asset_content_type: text/plain
+
+  update_version_badge:
+
+    runs-on: ubuntu-latest
+
+    needs: build_unix
+
+    steps:
+    - name: Create Version Badge
+      uses: schneegans/dynamic-badges-action@v1.0.0
+      with:
+        auth: ${{ secrets.GIST_TOKEN }}
+        gistID: c69cb23c3c5b3316248e52022790aa57
+        filename: version.json
+        label: Version
+        message: ${{ needs.build_unix.outputs.ytdlc_version }}

.github/workflows/ci.yml

@@ -3,14 +3,14 @@ on: [push]
 jobs:
   tests:
     name: Tests
+    if: "!contains(github.event.head_commit.message, 'skip ci')"
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: true
       matrix:
-        os: [ubuntu-latest]
+        os: [ubuntu-18.04]
         # TODO: python 2.6
-        # 3.3, 3.4 are not running
-        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
+        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
         python-impl: [cpython]
         ytdl-test-set: [core, download]
         run-tests-ext: [sh]

.github/workflows/quick-test.yml

@@ -3,6 +3,7 @@ on: [push]
 jobs:
   tests:
     name: Core Tests
+    if: "!contains(github.event.head_commit.message, 'skip ci all')"
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
@@ -18,6 +19,7 @@ jobs:
       run: ./devscripts/run_tests.sh
   flake8:
     name: Linter
+    if: "!contains(github.event.head_commit.message, 'skip ci all')"
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2

AUTHORS-Fork

@@ -1,8 +1,17 @@
-pukkandan
+pukkandan (owner)
 h-h-h-h
 pauldubois98
 nixxo
 GreyAlien502
 kyuyeunk
 siikamiika
-jbruchon
+jbruchon
+alexmerkel
+glenn-slayden
+Unrud
+wporr
+mariuszskon
+ohnonot
+samiksome
+alxnull
+FelixFrog

Changelog.md

@@ -1,11 +1,54 @@
 # Changelog
 
+<!--
+# Instuctions for creating release
 
-### 2020.01.08
-* **Merge youtube-dl:** Upto [2020.01.08](https://github.com/ytdl-org/youtube-dl/commit/bf6a74c620bd4d5726503c5302906bb36b009026)
+* Run `make doc`
+* Update Changelog.md and Authors-Fork
+* Commit to master as `Release <version>`
+* Push to origin/release - build task will now run
+* Update version.py and run `make issuetemplates`
+* Commit to master as `[version] update`
+* Push to origin/master
+* Update changelog in /releases
+
+-->
+
+
+### 2021.01.12
+* [roosterteeth.com] Add subtitle support by @samiksome
+* Added `--force-overwrites` by @alxnull
+* Changed fork name to `yt-dlp`
+* Fix typos by @FelixFrog
+* [ci] Option to skip
+* [changelog] Added unreleased changes in blackjack4494/yt-dlc
+
+
+### 2021.01.10
+* [archive.org] Fix extractor and add support for audio and playlists by @wporr
+* [Animelab] Added by @mariuszskon
+* [youtube:search] Fix view_count by @ohnonot
+* [youtube] Show if video is embeddable in info
+* Update version badge automatically in README
+* Enable `test_youtube_search_matching`
+* Create `to_screen` and similar functions in postprocessor/common
+
+
+### 2021.01.09
+* [youtube] Fix bug in automatic caption extraction
+* Add `post_hooks` to YoutubeDL by @alexmerkel
+* Batch file enumeration improvements by @glenn-slayden
+* Stop immediately when reaching `--max-downloads` by @glenn-slayden
+* Fix incorrect ANSI sequence for restoring console-window title by @glenn-slayden
+* Kill child processes when yt-dlc is killed by @Unrud
+
+
+### 2021.01.08
+* **Merge youtube-dl:** Upto [2021.01.08](https://github.com/ytdl-org/youtube-dl/commit/bf6a74c620bd4d5726503c5302906bb36b009026)
     * Extractor stitcher ([1](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc), [2](https://github.com/ytdl-org/youtube-dl/commit/a563c97c5cddf55f8989ed7ea8314ef78e30107f)) have not been merged
 * Moved changelog to seperate file
 
+
 ### 2021.01.07-1
 * [Akamai] fix by @nixxo
 * [Tiktok] merge youtube-dl tiktok extractor by @GreyAlien502
@@ -16,11 +59,13 @@
 * Deprecated `--sponskrub-args`. The same can now be done using `--postprocessor-args "sponskrub:<args>"`
 * [CI] Split tests into core-test and full-test
 
+
 ### 2021.01.07
 * Removed priority of `av01` codec in `-S` since most devices don't support it yet
 * Added `duration_string` to be used in `--output`
 * Created First Release
 
+
 ### 2021.01.05-1
 * **Changed defaults:**
     * Enabled `--ignore`
@@ -31,6 +76,7 @@
     * Changed default output template to `%(title)s [%(id)s].%(ext)s`
     * Enabled `--list-formats-as-table`
 
+
 ### 2021.01.05
 * **Format Sort:** Added `--format-sort` (`-S`), `--format-sort-force` (`--S-force`) - See [Sorting Formats](README.md#sorting-formats) for details
 * **Format Selection:** See [Format Selection](README.md#format-selection) for details
@@ -42,12 +88,42 @@
 * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details
 * Added `--force-download-archive` (`--force-write-archive`) by by h-h-h-h
 * Added `--list-formats-as-table`,  `--list-formats-old`
-* **Negative Options:** Makes it possible to negate boolean options by adding a `no-` to the switch
+* **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file
     * Added `--no-ignore-dynamic-mpd`, `--no-allow-dynamic-mpd`, `--allow-dynamic-mpd`, `--youtube-include-hls-manifest`, `--no-youtube-include-hls-manifest`, `--no-youtube-skip-hls-manifest`, `--no-download`, `--no-download-archive`, `--resize-buffer`, `--part`, `--mtime`, `--no-keep-fragments`, `--no-cookies`, `--no-write-annotations`, `--no-write-info-json`, `--no-write-description`, `--no-write-thumbnail`, `--youtube-include-dash-manifest`, `--post-overwrites`, `--no-keep-video`, `--no-embed-subs`, `--no-embed-thumbnail`, `--no-add-metadata`, `--no-include-ads`, `--no-write-sub`, `--no-write-auto-sub`, `--no-playlist-reverse`, `--no-restrict-filenames`, `--youtube-include-dash-manifest`, `--no-format-sort-force`, `--flat-videos`, `--no-list-formats-as-table`, `--no-sponskrub`, `--no-sponskrub-cut`, `--no-sponskrub-force`
     * Renamed: `--write-subs`, `--no-write-subs`, `--no-write-auto-subs`, `--write-auto-subs`. Note that these can still be used without the ending "s"
 * Relaxed validation for format filters so that any arbitrary field can be used
 * Fix for embedding thumbnail in mp3 by @pauldubois98
 * Make Twitch Video ID output from Playlist and VOD extractor same. This is only a temporary fix
-* **Merge youtube-dl:** Upto [2020.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
+* **Merge youtube-dl:** Upto [2021.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
     * Extractors [tiktok](https://github.com/ytdl-org/youtube-dl/commit/fb626c05867deab04425bad0c0b16b55473841a2) and [hotstar](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc) have not been merged
 * Cleaned up the fork for public use
+
+
+### Unreleased changes in [blackjack4494/yt-dlc](https://github.com/blackjack4494/yt-dlc)
+* Updated to youtube-dl release 2020.11.26
+* [youtube]
+    * Implemented all Youtube Feeds (ytfav, ytwatchlater, ytsubs, ythistory, ytrec) and SearchURL
+    * Fix ytsearch not returning results sometimes due to promoted content
+    * Temporary fix for automatic captions - disable json3
+    * Fix some improper Youtube URLs
+    * Redirect channel home to /video
+    * Print youtube's warning message
+    * Multiple pages are handled better for feeds
+* Add --break-on-existing by @gergesh
+* Pre-check video IDs in the archive before downloading
+* [bitwave.tv] New extractor
+* [Gedi] Add extractor
+* [Rcs] Add new extractor
+* [skyit] Add support for multiple Sky Italia website and removed old skyitalia extractor
+* [france.tv] Fix thumbnail URL
+* [ina] support mobile links
+* [instagram] Fix extractor
+* [itv] BTCC new pages' URL update (articles instead of races)
+* [SouthparkDe] Support for English URLs
+* [spreaker] fix SpreakerShowIE test URL
+* [Vlive] Fix playlist handling when downloading a channel
+* [generic] Detect embedded bitchute videos
+* [generic] Extract embedded youtube and twitter videos
+* [ffmpeg] Ensure all streams are copied
+* Fix for os.rename error when embedding thumbnail to video in a different drive
+* make_win.bat: don't use UPX to pack vcruntime140.dll

Makefile

@@ -10,7 +10,8 @@ PREFIX ?= /usr/local
 BINDIR ?= $(PREFIX)/bin
 MANDIR ?= $(PREFIX)/man
 SHAREDIR ?= $(PREFIX)/share
-PYTHON ?= /usr/bin/env python
+# make_supportedsites.py doesnot work correctly in python2
+PYTHON ?= /usr/bin/env python3
 
 # set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
 SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi)
@@ -46,6 +47,7 @@ offlinetest: codetest
 		--exclude test_age_restriction.py \
 		--exclude test_download.py \
 		--exclude test_iqiyi_sdk_interpreter.py \
+		--exclude test_overwrites.py \
 		--exclude test_socks.py \
 		--exclude test_subtitles.py \
 		--exclude test_write_annotations.py \

README.md

@@ -1,9 +1,12 @@
-[![Release Version](https://img.shields.io/badge/Release-2021.01.09-brightgreen)](https://github.com/pukkandan/yt-dlc/releases/latest)
+# YT-DLP
+
+<!-- See: https://github.com/marketplace/actions/dynamic-badges -->
+[![Release Version](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/pukkandan/c69cb23c3c5b3316248e52022790aa57/raw/version.json&color=brightgreen)](https://github.com/pukkandan/yt-dlc/releases/latest)
 [![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](https://github.com/pukkandan/yt-dlc/blob/master/LICENSE)
 [![Core Status](https://github.com/pukkandan/yt-dlc/workflows/Core%20Test/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlc/actions?query=workflow%3ACore)
 [![CI Status](https://github.com/pukkandan/yt-dlc/workflows/Full%20Test/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlc/actions?query=workflow%3AFull)
 
-youtube-dlc - download videos from youtube.com and many other [video platforms](docs/supportedsites.md)
+A command-line program to download videos from youtube.com and many other [video platforms](docs/supportedsites.md)
 
 This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which is inturn a fork of [youtube-dl](https://github.com/ytdl-org/youtube-dl)
 
@@ -11,7 +14,6 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
 * [INSTALLATION](#installation)
     * [UPDATE](#update)
     * [COMPILE](#compile)
-* [YOUTUBE-DLC](#youtube-dlc)
 * [DESCRIPTION](#description)
 * [OPTIONS](#options)
     * [Network Options](#network-options)
@@ -44,19 +46,33 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
 
 
 # NEW FEATURES
-The major new features are:
+The major new features from the latest release of [blackjack4494/yt-dlc](https://github.com/blackjack4494/yt-dlc) are:
 
-* **[SponSkrub Integration](#sponSkrub-options-sponsorblock)** - You can use [SponSkrub](https://github.com/faissaloo/SponSkrub) to mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
+* **[SponSkrub Integration](#sponSkrub-options-sponsorblock)**: You can use [SponSkrub](https://github.com/faissaloo/SponSkrub) to mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
 
-* **[Format Sorting](#sorting-format)** - The default format sorting options have been changed so that higher resolution and better codecs will be now prefered instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
+* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now prefered instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* Merged with youtube-dl **v2020.01.08** - You get the new features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494)
+* Merged with youtube-dl **v2020.01.08**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494)
 
-* **New options** - `--list-formats-as-table`, `--write-link`, `--force-download-archive` etc
+* **Youtube improvements**:
+    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and support downloading multiple pages of content
+    * Youtube search works correctly (`ytsearch:`, `ytsearchdate:`) along with Search URLs
+    * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
+
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv
+
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina
+
+* **New options**: `--list-formats-as-table`, `--write-link`, `--force-download-archive` etc
 
 and many other features and patches. See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlc/commits) for the full list of changes
 
 
+**PS**: Some of these changes are already in youtube-dlc, but are still unreleased. See [this](changelog.md#unreleased-changes-in-blackjack4494yt-dlc) for details
+
+If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the amount of changes are very large. Compare [options](#options) and [supported sites](docs/supportedsites.md) with youtube-dl's to get an idea of the massive number of features/patches [youtube-dlc](https://github.com/blackjack4494/yt-dlc) has accumulated.
+
+
 # INSTALLATION
 
 To use the latest version, simply download and run the [latest release](https://github.com/pukkandan/yt-dlc/releases/latest).
@@ -92,9 +108,9 @@ Then simply type this
 
 
 # DESCRIPTION
-**youtube-dlc** is a command-line program to download videos from YouTube.com and a few more sites. It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
+**youtube-dlc** is a command-line program to download videos from youtube.com many other [video platforms](docs/supportedsites.md). It requires the Python interpreter, version 2.6, 2.7, or 3.2+, and it is not platform specific. It should work on your Unix box, on Windows or on macOS. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
 
-    youtube-dlc [OPTIONS] URL [URL...]
+    youtube-dlc [OPTIONS] [--] URL [URL...]
 
 
 # OPTIONS
@@ -303,7 +319,11 @@ Then simply type this
                                      filenames
     --no-restrict-filenames          Allow Unicode characters, "&" and spaces in
                                      filenames (default)
-    -w, --no-overwrites              Do not overwrite files
+    -w, --no-overwrites              Do not overwrite any files
+    --force-overwrites               Overwrite all video and metadata files.
+                                     This option includes --no-continue
+    --no-force-overwrites            Do not overwrite the video, but overwrite
+                                     related files (default)
     -c, --continue                   Resume partially downloaded files (default)
     --no-continue                    Restart download of partially downloaded
                                      files from beginning
@@ -396,7 +416,8 @@ Then simply type this
                                      files in the current directory to debug
                                      problems
     --print-traffic                  Display sent and read HTTP traffic
-    -C, --call-home                  Contact the youtube-dlc server for debugging
+    -C, --call-home                  [Broken] Contact the youtube-dlc server for
+                                     debugging
     --no-call-home                   Do not contact the youtube-dlc server for
                                      debugging (default)
 

devscripts/run_tests.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 # Keep this list in sync with the `offlinetest` target in Makefile
-DOWNLOAD_TESTS="age_restriction|download|iqiyi_sdk_interpreter|socks|subtitles|write_annotations|youtube_lists|youtube_signature|post_hooks"
+DOWNLOAD_TESTS="age_restriction|download|iqiyi_sdk_interpreter|overwrites|socks|subtitles|write_annotations|youtube_lists|youtube_signature|post_hooks"
 
 test_set=""
 multiprocess_args=""

docs/supportedsites.md

@@ -48,6 +48,8 @@
  - **AMCNetworks**
  - **AmericasTestKitchen**
  - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
+ - **AnimeLab**
+ - **AnimeLabShows**
  - **AnimeOnDemand**
  - **Anvato**
  - **aol.com**
@@ -58,7 +60,7 @@
  - **ApplePodcasts**
  - **appletrailers**
  - **appletrailers:section**
- - **archive.org**: archive.org videos
+ - **archive.org**: archive.org video and audio
  - **ArcPublishing**
  - **ARD**
  - **ARD:mediathek**

pyinst.py

@@ -74,7 +74,7 @@ version_file = VSVersionInfo(
                         ),
                         StringStruct("OriginalFilename", "youtube-dlc.exe"),
                         StringStruct("ProductName", "Youtube-dlc"),
-                        StringStruct("ProductVersion", version + " | git.io/JUGsM"),
+                        StringStruct("ProductVersion", version + " | git.io/JLh7K"),
                     ],
                 )
             ]

test/parameters.json

@@ -14,7 +14,7 @@
     "logtostderr": false, 
     "matchtitle": null, 
     "max_downloads": null, 
-    "nooverwrites": false, 
+    "overwrites": null, 
     "nopart": false, 
     "noprogress": false, 
     "outtmpl": "%(id)s.%(ext)s", 

test/test_all_urls.py

@@ -69,9 +69,9 @@ class TestAllURLsMatching(unittest.TestCase):
         self.assertMatch('https://www.youtube.com/feed/watch_later', ['youtube:tab'])
         self.assertMatch('https://www.youtube.com/feed/subscriptions', ['youtube:tab'])
 
-    # def test_youtube_search_matching(self):
-    #     self.assertMatch('http://www.youtube.com/results?search_query=making+mustard', ['youtube:search_url'])
-    #     self.assertMatch('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video', ['youtube:search_url'])
+    def test_youtube_search_matching(self):
+        self.assertMatch('http://www.youtube.com/results?search_query=making+mustard', ['youtube:search_url'])
+        self.assertMatch('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video', ['youtube:search_url'])
 
     def test_youtube_extract(self):
         assertExtractId = lambda url, id: self.assertEqual(YoutubeIE.extract_id(url), id)

test/test_overwrites.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python
+from __future__ import unicode_literals
+
+import os
+from os.path import join
+import subprocess
+import sys
+import unittest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from test.helper import try_rm
+
+
+root_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+download_file = join(root_dir, 'test.webm')
+
+
+class TestOverwrites(unittest.TestCase):
+    def setUp(self):
+        # create an empty file
+        open(download_file, 'a').close()
+
+    def test_default_overwrites(self):
+        outp = subprocess.Popen(
+            [
+                sys.executable, 'youtube_dlc/__main__.py',
+                '-o', 'test.webm',
+                'https://www.youtube.com/watch?v=jNQXAC9IVRw'
+            ], cwd=root_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        sout, serr = outp.communicate()
+        self.assertTrue(b'has already been downloaded' in sout)
+        # if the file has no content, it has not been redownloaded
+        self.assertTrue(os.path.getsize(download_file) < 1)
+
+    def test_yes_overwrites(self):
+        outp = subprocess.Popen(
+            [
+                sys.executable, 'youtube_dlc/__main__.py', '--yes-overwrites',
+                '-o', 'test.webm',
+                'https://www.youtube.com/watch?v=jNQXAC9IVRw'
+            ], cwd=root_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        sout, serr = outp.communicate()
+        self.assertTrue(b'has already been downloaded' not in sout)
+        # if the file has no content, it has not been redownloaded
+        self.assertTrue(os.path.getsize(download_file) > 1)
+
+    def tearDown(self):
+        try_rm(join(root_dir, 'test.webm'))
+
+
+if __name__ == '__main__':
+    unittest.main()

youtube-dlc.cmd

@@ -1 +1 @@
-py "%~dp0\youtube_dl\__main__.py"
+py "%~dp0youtube_dl\__main__.py"

youtube_dlc/YoutubeDL.py

@@ -179,9 +179,11 @@ class YoutubeDL(object):
     outtmpl:           Template for output names.
     restrictfilenames: Do not allow "&" and spaces in file names.
     trim_file_name:    Limit length of filename (extension excluded).
-    ignoreerrors:      Do not stop on download errors. (Default False when running youtube-dlc, but True when directly accessing YoutubeDL class)
+    ignoreerrors:      Do not stop on download errors. (Default True when running youtube-dlc, but False when directly accessing YoutubeDL class)
     force_generic_extractor: Force downloader to use the generic extractor
-    nooverwrites:      Prevent overwriting files.
+    overwrites:        Overwrite all video and metadata files if True,
+                       overwrite only non-video files if None
+                       and don't overwrite any file if False
     playliststart:     Playlist item to start at.
     playlistend:       Playlist item to end at.
     playlist_items:    Specific indices of playlist to download.
@@ -686,6 +688,13 @@ class YoutubeDL(object):
         except UnicodeEncodeError:
             self.to_screen('[download] The file has already been downloaded')
 
+    def report_file_delete(self, file_name):
+        """Report that existing file will be deleted."""
+        try:
+            self.to_screen('Deleting already existent file %s' % file_name)
+        except UnicodeEncodeError:
+            self.to_screen('Deleting already existent file')
+
     def prepare_filename(self, info_dict):
         """Generate the output filename."""
         try:
@@ -1898,7 +1907,7 @@ class YoutubeDL(object):
 
         if self.params.get('writedescription', False):
             descfn = replace_extension(filename, 'description', info_dict.get('ext'))
-            if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(descfn)):
+            if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(descfn)):
                 self.to_screen('[info] Video description is already present')
             elif info_dict.get('description') is None:
                 self.report_warning('There\'s no description to write.')
@@ -1913,7 +1922,7 @@ class YoutubeDL(object):
 
         if self.params.get('writeannotations', False):
             annofn = replace_extension(filename, 'annotations.xml', info_dict.get('ext'))
-            if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(annofn)):
+            if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(annofn)):
                 self.to_screen('[info] Video annotations are already present')
             elif not info_dict.get('annotations'):
                 self.report_warning('There are no annotations to write.')
@@ -1947,7 +1956,7 @@ class YoutubeDL(object):
             for sub_lang, sub_info in subtitles.items():
                 sub_format = sub_info['ext']
                 sub_filename = subtitles_filename(filename, sub_lang, sub_format, info_dict.get('ext'))
-                if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(sub_filename)):
+                if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(sub_filename)):
                     self.to_screen('[info] Video subtitle %s.%s is already present' % (sub_lang, sub_format))
                 else:
                     self.to_screen('[info] Writing video subtitles to: ' + sub_filename)
@@ -2002,7 +2011,7 @@ class YoutubeDL(object):
 
         if self.params.get('writeinfojson', False):
             infofn = replace_extension(filename, 'info.json', info_dict.get('ext'))
-            if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(infofn)):
+            if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(infofn)):
                 self.to_screen('[info] Video description metadata is already present')
             else:
                 self.to_screen('[info] Writing video description metadata as JSON to: ' + infofn)
@@ -2110,11 +2119,15 @@ class YoutubeDL(object):
                             'Requested formats are incompatible for merge and will be merged into mkv.')
                     # Ensure filename always has a correct extension for successful merge
                     filename = '%s.%s' % (filename_wo_ext, info_dict['ext'])
-                    if os.path.exists(encodeFilename(filename)):
+                    file_exists = os.path.exists(encodeFilename(filename))
+                    if not self.params.get('overwrites', False) and file_exists:
                         self.to_screen(
                             '[download] %s has already been downloaded and '
                             'merged' % filename)
                     else:
+                        if file_exists:
+                            self.report_file_delete(filename)
+                            os.remove(encodeFilename(filename))
                         for f in requested_formats:
                             new_info = dict(info_dict)
                             new_info.update(f)
@@ -2131,6 +2144,11 @@ class YoutubeDL(object):
                         # Even if there were no downloads, it is being merged only now
                         info_dict['__real_download'] = True
                 else:
+                    # Delete existing file with --yes-overwrites
+                    if self.params.get('overwrites', False):
+                        if os.path.exists(encodeFilename(filename)):
+                            self.report_file_delete(filename)
+                            os.remove(encodeFilename(filename))
                     # Just a single file
                     success, real_download = dl(filename, info_dict)
                     info_dict['__real_download'] = real_download
@@ -2514,7 +2532,7 @@ class YoutubeDL(object):
                 self.get_encoding()))
         write_string(encoding_str, encoding=None)
 
-        self._write_string('[debug] youtube-dlc version ' + __version__ + '\n')
+        self._write_string('[debug] yt-dlp version ' + __version__ + '\n')
         if _LAZY_LOADER:
             self._write_string('[debug] Lazy loading extractors enabled' + '\n')
         try:
@@ -2563,6 +2581,7 @@ class YoutubeDL(object):
         if self.params.get('call_home', False):
             ipaddr = self.urlopen('https://yt-dl.org/ip').read().decode('utf-8')
             self._write_string('[debug] Public IP address: %s\n' % ipaddr)
+            return
             latest_version = self.urlopen(
                 'https://yt-dl.org/latest/version').read().decode('utf-8')
             if version_tuple(latest_version) > version_tuple(__version__):
@@ -2660,7 +2679,7 @@ class YoutubeDL(object):
             thumb_display_id = '%s ' % t['id'] if len(thumbnails) > 1 else ''
             t['filename'] = thumb_filename = replace_extension(filename + suffix, thumb_ext, info_dict.get('ext'))
 
-            if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(thumb_filename)):
+            if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
                 self.to_screen('[%s] %s: Thumbnail %sis already present' %
                                (info_dict['extractor'], info_dict['id'], thumb_display_id))
             else:

youtube_dlc/__init__.py

@@ -176,6 +176,9 @@ def _real_main(argv=None):
         opts.max_sleep_interval = opts.sleep_interval
     if opts.ap_mso and opts.ap_mso not in MSO_INFO:
         parser.error('Unsupported TV Provider, use --ap-list-mso to get a list of supported TV Providers')
+    if opts.overwrites:
+        # --yes-overwrites implies --no-continue
+        opts.continue_dl = False
 
     def parse_retries(retries):
         if retries in ('inf', 'infinite'):
@@ -391,7 +394,7 @@ def _real_main(argv=None):
         'ignoreerrors': opts.ignoreerrors,
         'force_generic_extractor': opts.force_generic_extractor,
         'ratelimit': opts.ratelimit,
-        'nooverwrites': opts.nooverwrites,
+        'overwrites': opts.overwrites,
         'retries': opts.retries,
         'fragment_retries': opts.fragment_retries,
         'skip_unavailable_fragments': opts.skip_unavailable_fragments,

youtube_dlc/downloader/common.py

@@ -332,7 +332,7 @@ class FileDownloader(object):
         """
 
         nooverwrites_and_exists = (
-            self.params.get('nooverwrites', False)
+            not self.params.get('overwrites', True)
             and os.path.exists(encodeFilename(filename))
         )
 

youtube_dlc/extractor/adobepass.py

@@ -9,6 +9,7 @@ from .common import InfoExtractor
 from ..compat import (
     compat_kwargs,
     compat_urlparse,
+    compat_getpass
 )
 from ..utils import (
     unescapeHTML,
@@ -60,6 +61,10 @@ MSO_INFO = {
         'username_field': 'IDToken1',
         'password_field': 'IDToken2',
     },
+    'Philo': {
+        'name': 'Philo',
+        'username_field': 'ident'
+    },
     'Verizon': {
         'name': 'Verizon FiOS',
         'username_field': 'IDToken1',
@@ -1467,6 +1472,22 @@ class AdobePassIE(InfoExtractor):
                         mvpd_confirm_page, urlh = mvpd_confirm_page_res
                         if '<button class="submit" value="Resume">Resume</button>' in mvpd_confirm_page:
                             post_form(mvpd_confirm_page_res, 'Confirming Login')
+                elif mso_id == 'Philo':
+                    # Philo has very unique authentication method
+                    self._download_webpage(
+                        'https://idp.philo.com/auth/init/login_code', video_id, 'Requesting auth code', data=urlencode_postdata({
+                            'ident': username,
+                            'device': 'web',
+                            'send_confirm_link': False,
+                            'send_token': True
+                        }))
+                    philo_code = compat_getpass('Type auth code you have received [Return]: ')
+                    self._download_webpage(
+                        'https://idp.philo.com/auth/update/login_code', video_id, 'Submitting token', data=urlencode_postdata({
+                            'token': philo_code
+                        }))
+                    mvpd_confirm_page_res = self._download_webpage_handle('https://idp.philo.com/idp/submit', video_id, 'Confirming Philo Login')
+                    post_form(mvpd_confirm_page_res, 'Confirming Login')
                 elif mso_id == 'Verizon':
                     # In general, if you're connecting from a Verizon-assigned IP,
                     # you will not actually pass your credentials.

youtube_dlc/extractor/animelab.py

@@ -0,0 +1,285 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+from ..utils import (
+    ExtractorError,
+    urlencode_postdata,
+    int_or_none,
+    str_or_none,
+    determine_ext,
+)
+
+from ..compat import compat_HTTPError
+
+
+class AnimeLabBaseIE(InfoExtractor):
+    _LOGIN_REQUIRED = True
+    _LOGIN_URL = 'https://www.animelab.com/login'
+    _NETRC_MACHINE = 'animelab'
+
+    def _login(self):
+        def is_logged_in(login_webpage):
+            return 'Sign In' not in login_webpage
+
+        login_page = self._download_webpage(
+            self._LOGIN_URL, None, 'Downloading login page')
+
+        # Check if already logged in
+        if is_logged_in(login_page):
+            return
+
+        (username, password) = self._get_login_info()
+        if username is None and self._LOGIN_REQUIRED:
+            self.raise_login_required('Login is required to access any AnimeLab content')
+
+        login_form = {
+            'email': username,
+            'password': password,
+        }
+
+        try:
+            response = self._download_webpage(
+                self._LOGIN_URL, None, 'Logging in', 'Wrong login info',
+                data=urlencode_postdata(login_form),
+                headers={'Content-Type': 'application/x-www-form-urlencoded'})
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 400:
+                raise ExtractorError('Unable to log in (wrong credentials?)', expected=True)
+            else:
+                raise
+
+        # if login was successful
+        if is_logged_in(response):
+            return
+
+        raise ExtractorError('Unable to login (cannot verify if logged in)')
+
+    def _real_initialize(self):
+        self._login()
+
+
+class AnimeLabIE(AnimeLabBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?animelab\.com/player/(?P<id>[^/]+)'
+
+    # the following tests require authentication, but a free account will suffice
+    # just set 'usenetrc' to true in test/local_parameters.json if you use a .netrc file
+    # or you can set 'username' and 'password' there
+    # the tests also select a specific format so that the same video is downloaded
+    # regardless of whether the user is premium or not (needs testing on a premium account)
+    _TEST = {
+        'url': 'https://www.animelab.com/player/fullmetal-alchemist-brotherhood-episode-42',
+        'md5': '05bde4b91a5d1ff46ef5b94df05b0f7f',
+        'info_dict': {
+            'id': '383',
+            'ext': 'mp4',
+            'display_id': 'fullmetal-alchemist-brotherhood-episode-42',
+            'title': 'Fullmetal Alchemist: Brotherhood - Episode 42 - Signs of a Counteroffensive',
+            'description': 'md5:103eb61dd0a56d3dfc5dbf748e5e83f4',
+            'series': 'Fullmetal Alchemist: Brotherhood',
+            'episode': 'Signs of a Counteroffensive',
+            'episode_number': 42,
+            'duration': 1469,
+            'season': 'Season 1',
+            'season_number': 1,
+            'season_id': '38',
+        },
+        'params': {
+            'format': '[format_id=21711_yeshardsubbed_ja-JP][height=480]',
+        },
+        'skip': 'All AnimeLab content requires authentication',
+    }
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+
+        # unfortunately we can get different URLs for the same formats
+        # e.g. if we are using a "free" account so no dubs available
+        # (so _remove_duplicate_formats is not effective)
+        # so we use a dictionary as a workaround
+        formats = {}
+        for language_option_url in ('https://www.animelab.com/player/%s/subtitles',
+                                    'https://www.animelab.com/player/%s/dubbed'):
+            actual_url = language_option_url % display_id
+            webpage = self._download_webpage(actual_url, display_id, 'Downloading URL ' + actual_url)
+
+            video_collection = self._parse_json(self._search_regex(r'new\s+?AnimeLabApp\.VideoCollection\s*?\((.*?)\);', webpage, 'AnimeLab VideoCollection'), display_id)
+            position = int_or_none(self._search_regex(r'playlistPosition\s*?=\s*?(\d+)', webpage, 'Playlist Position'))
+
+            raw_data = video_collection[position]['videoEntry']
+
+            video_id = str_or_none(raw_data['id'])
+
+            # create a title from many sources (while grabbing other info)
+            # TODO use more fallback sources to get some of these
+            series = raw_data.get('showTitle')
+            video_type = raw_data.get('videoEntryType', {}).get('name')
+            episode_number = raw_data.get('episodeNumber')
+            episode_name = raw_data.get('name')
+
+            title_parts = (series, video_type, episode_number, episode_name)
+            if None not in title_parts:
+                title = '%s - %s %s - %s' % title_parts
+            else:
+                title = episode_name
+
+            description = raw_data.get('synopsis') or self._og_search_description(webpage, default=None)
+
+            duration = int_or_none(raw_data.get('duration'))
+
+            thumbnail_data = raw_data.get('images', [])
+            thumbnails = []
+            for thumbnail in thumbnail_data:
+                for instance in thumbnail['imageInstances']:
+                    image_data = instance.get('imageInfo', {})
+                    thumbnails.append({
+                        'id': str_or_none(image_data.get('id')),
+                        'url': image_data.get('fullPath'),
+                        'width': image_data.get('width'),
+                        'height': image_data.get('height'),
+                    })
+
+            season_data = raw_data.get('season', {}) or {}
+            season = str_or_none(season_data.get('name'))
+            season_number = int_or_none(season_data.get('seasonNumber'))
+            season_id = str_or_none(season_data.get('id'))
+
+            for video_data in raw_data['videoList']:
+                current_video_list = {}
+                current_video_list['language'] = video_data.get('language', {}).get('languageCode')
+
+                is_hardsubbed = video_data.get('hardSubbed')
+
+                for video_instance in video_data['videoInstances']:
+                    httpurl = video_instance.get('httpUrl')
+                    url = httpurl if httpurl else video_instance.get('rtmpUrl')
+                    if url is None:
+                        # this video format is unavailable to the user (not premium etc.)
+                        continue
+
+                    current_format = current_video_list.copy()
+
+                    format_id_parts = []
+
+                    format_id_parts.append(str_or_none(video_instance.get('id')))
+
+                    if is_hardsubbed is not None:
+                        if is_hardsubbed:
+                            format_id_parts.append('yeshardsubbed')
+                        else:
+                            format_id_parts.append('nothardsubbed')
+
+                    format_id_parts.append(current_format['language'])
+
+                    format_id = '_'.join([x for x in format_id_parts if x is not None])
+
+                    ext = determine_ext(url)
+                    if ext == 'm3u8':
+                        for format_ in self._extract_m3u8_formats(
+                                url, video_id, m3u8_id=format_id, fatal=False):
+                            formats[format_['format_id']] = format_
+                        continue
+                    elif ext == 'mpd':
+                        for format_ in self._extract_mpd_formats(
+                                url, video_id, mpd_id=format_id, fatal=False):
+                            formats[format_['format_id']] = format_
+                        continue
+
+                    current_format['url'] = url
+                    quality_data = video_instance.get('videoQuality')
+                    if quality_data:
+                        quality = quality_data.get('name') or quality_data.get('description')
+                    else:
+                        quality = None
+
+                    height = None
+                    if quality:
+                        height = int_or_none(self._search_regex(r'(\d+)p?$', quality, 'Video format height', default=None))
+
+                    if height is None:
+                        self.report_warning('Could not get height of video')
+                    else:
+                        current_format['height'] = height
+                    current_format['format_id'] = format_id
+
+                    formats[current_format['format_id']] = current_format
+
+        formats = list(formats.values())
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'display_id': display_id,
+            'title': title,
+            'description': description,
+            'series': series,
+            'episode': episode_name,
+            'episode_number': int_or_none(episode_number),
+            'thumbnails': thumbnails,
+            'duration': duration,
+            'formats': formats,
+            'season': season,
+            'season_number': season_number,
+            'season_id': season_id,
+        }
+
+
+class AnimeLabShowsIE(AnimeLabBaseIE):
+    _VALID_URL = r'https?://(?:www\.)?animelab\.com/shows/(?P<id>[^/]+)'
+
+    _TEST = {
+        'url': 'https://www.animelab.com/shows/attack-on-titan',
+        'info_dict': {
+            'id': '45',
+            'title': 'Attack on Titan',
+            'description': 'md5:989d95a2677e9309368d5cf39ba91469',
+        },
+        'playlist_count': 59,
+        'skip': 'All AnimeLab content requires authentication',
+    }
+
+    def _real_extract(self, url):
+        _BASE_URL = 'http://www.animelab.com'
+        _SHOWS_API_URL = '/api/videoentries/show/videos/'
+        display_id = self._match_id(url)
+
+        webpage = self._download_webpage(url, display_id, 'Downloading requested URL')
+
+        show_data_str = self._search_regex(r'({"id":.*}),\svideoEntry', webpage, 'AnimeLab show data')
+        show_data = self._parse_json(show_data_str, display_id)
+
+        show_id = str_or_none(show_data.get('id'))
+        title = show_data.get('name')
+        description = show_data.get('shortSynopsis') or show_data.get('longSynopsis')
+
+        entries = []
+        for season in show_data['seasons']:
+            season_id = season['id']
+            get_data = urlencode_postdata({
+                'seasonId': season_id,
+                'limit': 1000,
+            })
+            # despite using urlencode_postdata, we are sending a GET request
+            target_url = _BASE_URL + _SHOWS_API_URL + show_id + "?" + get_data.decode('utf-8')
+            response = self._download_webpage(
+                target_url,
+                None, 'Season id %s' % season_id)
+
+            season_data = self._parse_json(response, display_id)
+
+            for video_data in season_data['list']:
+                entries.append(self.url_result(
+                    _BASE_URL + '/player/' + video_data['slug'], 'AnimeLab',
+                    str_or_none(video_data.get('id')), video_data.get('name')
+                ))
+
+        return {
+            '_type': 'playlist',
+            'id': show_id,
+            'title': title,
+            'description': description,
+            'entries': entries,
+        }
+
+# TODO implement myqueue

youtube_dlc/extractor/archiveorg.py

@@ -1,27 +1,43 @@
 from __future__ import unicode_literals
 
+import re
+import json
+
 from .common import InfoExtractor
+from ..compat import compat_urllib_parse_unquote_plus
 from ..utils import (
+    KNOWN_EXTENSIONS,
+
+    extract_attributes,
     unified_strdate,
+    unified_timestamp,
     clean_html,
+    dict_get,
+    parse_duration,
+    int_or_none,
+    str_or_none,
+    merge_dicts,
 )
 
 
 class ArchiveOrgIE(InfoExtractor):
     IE_NAME = 'archive.org'
-    IE_DESC = 'archive.org videos'
-    _VALID_URL = r'https?://(?:www\.)?archive\.org/(?:details|embed)/(?P<id>[^/?#]+)(?:[?].*)?$'
+    IE_DESC = 'archive.org video and audio'
+    _VALID_URL = r'https?://(?:www\.)?archive\.org/(?:details|embed)/(?P<id>[^?#]+)(?:[?].*)?$'
     _TESTS = [{
         'url': 'http://archive.org/details/XD300-23_68HighlightsAResearchCntAugHumanIntellect',
         'md5': '8af1d4cf447933ed3c7f4871162602db',
         'info_dict': {
             'id': 'XD300-23_68HighlightsAResearchCntAugHumanIntellect',
-            'ext': 'ogg',
+            'ext': 'ogv',
             'title': '1968 Demo - FJCC Conference Presentation Reel #1',
             'description': 'md5:da45c349df039f1cc8075268eb1b5c25',
-            'upload_date': '19681210',
-            'uploader': 'SRI International'
-        }
+            'release_date': '19681210',
+            'timestamp': 1268695290,
+            'upload_date': '20100315',
+            'creator': 'SRI International',
+            'uploader': 'laura@archive.org',
+        },
     }, {
         'url': 'https://archive.org/details/Cops1922',
         'md5': '0869000b4ce265e8ca62738b336b268a',
@@ -29,37 +45,199 @@ class ArchiveOrgIE(InfoExtractor):
             'id': 'Cops1922',
             'ext': 'mp4',
             'title': 'Buster Keaton\'s "Cops" (1922)',
-            'description': 'md5:89e7c77bf5d965dd5c0372cfb49470f6',
-        }
+            'description': 'md5:43a603fd6c5b4b90d12a96b921212b9c',
+            'uploader': 'yorkmba99@hotmail.com',
+            'timestamp': 1387699629,
+            'upload_date': "20131222",
+        },
     }, {
         'url': 'http://archive.org/embed/XD300-23_68HighlightsAResearchCntAugHumanIntellect',
         'only_matching': True,
+    }, {
+        'url': 'https://archive.org/details/Election_Ads',
+        'md5': '284180e857160cf866358700bab668a3',
+        'info_dict': {
+            'id': 'Election_Ads/Commercial-JFK1960ElectionAdCampaignJingle.mpg',
+            'title': 'Commercial-JFK1960ElectionAdCampaignJingle.mpg',
+            'ext': 'mp4',
+        },
+    }, {
+        'url': 'https://archive.org/details/Election_Ads/Commercial-Nixon1960ElectionAdToughonDefense.mpg',
+        'md5': '7915213ef02559b5501fe630e1a53f59',
+        'info_dict': {
+            'id': 'Election_Ads/Commercial-Nixon1960ElectionAdToughonDefense.mpg',
+            'title': 'Commercial-Nixon1960ElectionAdToughonDefense.mpg',
+            'ext': 'mp4',
+            'timestamp': 1205588045,
+            'uploader': 'mikedavisstripmaster@yahoo.com',
+            'description': '1960 Presidential Campaign Election Commercials John F Kennedy, Richard M Nixon',
+            'upload_date': '20080315',
+        },
+    }, {
+        'url': 'https://archive.org/details/gd1977-05-08.shure57.stevenson.29303.flac16',
+        'md5': '7d07ffb42aba6537c28e053efa4b54c9',
+        'info_dict': {
+            'id': 'gd1977-05-08.shure57.stevenson.29303.flac16/gd1977-05-08d01t01.flac',
+            'title': 'Turning',
+            'ext': 'flac',
+        },
+    }, {
+        'url': 'https://archive.org/details/gd1977-05-08.shure57.stevenson.29303.flac16/gd1977-05-08d01t07.flac',
+        'md5': 'a07cd8c6ab4ee1560f8a0021717130f3',
+        'info_dict': {
+            'id': 'gd1977-05-08.shure57.stevenson.29303.flac16/gd1977-05-08d01t07.flac',
+            'title': 'Deal',
+            'ext': 'flac',
+            'timestamp': 1205895624,
+            'uploader': 'mvernon54@yahoo.com',
+            'description': 'md5:6a31f1996db0aa0fc9da6d6e708a1bb0',
+            'upload_date': '20080319',
+            'location': 'Barton Hall - Cornell University',
+        },
+    }, {
+        'url': 'https://archive.org/details/lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik',
+        'md5': '7cb019baa9b332e82ea7c10403acd180',
+        'info_dict': {
+            'id': 'lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik/disc1/01.01. Bells Of Rostov.mp3',
+            'title': 'Bells Of Rostov',
+            'ext': 'mp3',
+        },
+    }, {
+        'url': 'https://archive.org/details/lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik/disc1/02.02.+Song+And+Chorus+In+The+Polovetsian+Camp+From+%22Prince+Igor%22+(Act+2%2C+Scene+1).mp3',
+        'md5': '1d0aabe03edca83ca58d9ed3b493a3c3',
+        'info_dict': {
+            'id': 'lp_the-music-of-russia_various-artists-a-askaryan-alexander-melik/disc1/02.02. Song And Chorus In The Polovetsian Camp From "Prince Igor" (Act 2, Scene 1).mp3',
+            'title': 'Song And Chorus In The Polovetsian Camp From "Prince Igor" (Act 2, Scene 1)',
+            'ext': 'mp3',
+            'timestamp': 1569662587,
+            'uploader': 'associate-joygen-odiongan@archive.org',
+            'description': 'md5:012b2d668ae753be36896f343d12a236',
+            'upload_date': '20190928',
+        },
     }]
 
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(
-            'http://archive.org/embed/' + video_id, video_id)
-        jwplayer_playlist = self._parse_json(self._search_regex(
-            r"(?s)Play\('[^']+'\s*,\s*(\[.+\])\s*,\s*{.*?}\)",
-            webpage, 'jwplayer playlist'), video_id)
-        info = self._parse_jwplayer_data(
-            {'playlist': jwplayer_playlist}, video_id, base_url=url)
+    @staticmethod
+    def _playlist_data(webpage):
+        element = re.findall(r'''(?xs)
+            <input
+            (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]*|="[^"]*"|='[^']*'|))*?
+            \s+class=['"]?js-play8-playlist['"]?
+            (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]*|="[^"]*"|='[^']*'|))*?
+            \s*/>
+        ''', webpage)[0]
 
-        def get_optional(metadata, field):
-            return metadata.get(field, [None])[0]
+        return json.loads(extract_attributes(element)['value'])
+
+    def _real_extract(self, url):
+        video_id = compat_urllib_parse_unquote_plus(self._match_id(url))
+        identifier, entry_id = (video_id.split('/', 1) + [None])[:2]
+
+        # Archive.org metadata API doesn't clearly demarcate playlist entries
+        # or subtitle tracks, so we get them from the embeddable player.
+        embed_page = self._download_webpage(
+            'https://archive.org/embed/' + identifier, identifier)
+        playlist = self._playlist_data(embed_page)
+
+        entries = {}
+        for p in playlist:
+            # If the user specified a playlist entry in the URL, ignore the
+            # rest of the playlist.
+            if entry_id and p['orig'] != entry_id:
+                continue
+
+            entries[p['orig']] = {
+                'formats': [],
+                'thumbnails': [],
+                'artist': p.get('artist'),
+                'track': p.get('title'),
+                'subtitles': {}}
+
+            for track in p.get('tracks', []):
+                if track['kind'] != 'subtitles':
+                    continue
+
+                entries[p['orig']][track['label']] = {
+                    'url': 'https://archive.org/' + track['file'].lstrip('/')}
 
         metadata = self._download_json(
-            'http://archive.org/details/' + video_id, video_id, query={
-                'output': 'json',
-            })['metadata']
-        info.update({
-            'title': get_optional(metadata, 'title') or info.get('title'),
-            'description': clean_html(get_optional(metadata, 'description')),
-        })
-        if info.get('_type') != 'playlist':
-            info.update({
-                'uploader': get_optional(metadata, 'creator'),
-                'upload_date': unified_strdate(get_optional(metadata, 'date')),
-            })
+            'http://archive.org/metadata/' + identifier, identifier)
+        m = metadata['metadata']
+        identifier = m['identifier']
+
+        info = {
+            'id': identifier,
+            'title': m['title'],
+            'description': clean_html(m.get('description')),
+            'uploader': dict_get(m, ['uploader', 'adder']),
+            'creator': m.get('creator'),
+            'license': m.get('licenseurl'),
+            'release_date': unified_strdate(m.get('date')),
+            'timestamp': unified_timestamp(dict_get(m, ['publicdate', 'addeddate'])),
+            'webpage_url': 'https://archive.org/details/' + identifier,
+            'location': m.get('venue'),
+            'release_year': int_or_none(m.get('year'))}
+
+        for f in metadata['files']:
+            if f['name'] in entries:
+                entries[f['name']] = merge_dicts(entries[f['name']], {
+                    'id': identifier + '/' + f['name'],
+                    'title': f.get('title') or f['name'],
+                    'display_id': f['name'],
+                    'description': clean_html(f.get('description')),
+                    'creator': f.get('creator'),
+                    'duration': parse_duration(f.get('length')),
+                    'track_number': int_or_none(f.get('track')),
+                    'album': f.get('album'),
+                    'discnumber': int_or_none(f.get('disc')),
+                    'release_year': int_or_none(f.get('year'))})
+                entry = entries[f['name']]
+            elif f.get('original') in entries:
+                entry = entries[f['original']]
+            else:
+                continue
+
+            if f.get('format') == 'Thumbnail':
+                entry['thumbnails'].append({
+                    'id': f['name'],
+                    'url': 'https://archive.org/download/' + identifier + '/' + f['name'],
+                    'width': int_or_none(f.get('width')),
+                    'height': int_or_none(f.get('width')),
+                    'filesize': int_or_none(f.get('size'))})
+
+            extension = (f['name'].rsplit('.', 1) + [None])[1]
+            if extension in KNOWN_EXTENSIONS:
+                entry['formats'].append({
+                    'url': 'https://archive.org/download/' + identifier + '/' + f['name'],
+                    'format': f.get('format'),
+                    'width': int_or_none(f.get('width')),
+                    'height': int_or_none(f.get('height')),
+                    'filesize': int_or_none(f.get('size')),
+                    'protocol': 'https'})
+
+        # Sort available formats by filesize
+        for entry in entries.values():
+            entry['formats'] = list(sorted(entry['formats'], key=lambda x: x.get('filesize', -1)))
+
+        if len(entries) == 1:
+            # If there's only one item, use it as the main info dict
+            only_video = entries[list(entries.keys())[0]]
+            if entry_id:
+                info = merge_dicts(only_video, info)
+            else:
+                info = merge_dicts(info, only_video)
+        else:
+            # Otherwise, we have a playlist.
+            info['_type'] = 'playlist'
+            info['entries'] = list(entries.values())
+
+        if metadata.get('reviews'):
+            info['comments'] = []
+            for review in metadata['reviews']:
+                info['comments'].append({
+                    'id': review.get('review_id'),
+                    'author': review.get('reviewer'),
+                    'text': str_or_none(review.get('reviewtitle'), '') + '\n\n' + review.get('reviewbody'),
+                    'timestamp': unified_timestamp(review.get('createdate')),
+                    'parent': 'root'})
+
         return info

youtube_dlc/extractor/extractors.py

@@ -46,6 +46,10 @@ from .alura import (
     AluraCourseIE
 )
 from .amcnetworks import AMCNetworksIE
+from .animelab import (
+    AnimeLabIE,
+    AnimeLabShowsIE,
+)
 from .americastestkitchen import AmericasTestKitchenIE
 from .animeondemand import AnimeOnDemandIE
 from .anvato import AnvatoIE

youtube_dlc/extractor/roosterteeth.py

@@ -86,9 +86,11 @@ class RoosterTeethIE(InfoExtractor):
         api_episode_url = self._EPISODE_BASE_URL + display_id
 
         try:
-            m3u8_url = self._download_json(
+            video_data = self._download_json(
                 api_episode_url + '/videos', display_id,
-                'Downloading video JSON metadata')['data'][0]['attributes']['url']
+                'Downloading video JSON metadata')['data'][0]
+            m3u8_url = video_data['attributes']['url']
+            subtitle_m3u8_url = video_data['links']['download']
         except ExtractorError as e:
             if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
                 if self._parse_json(e.cause.read().decode(), display_id).get('access') is False:
@@ -119,6 +121,33 @@ class RoosterTeethIE(InfoExtractor):
                             'url': img_url,
                         })
 
+        subtitles = {}
+        res = self._download_webpage_handle(
+            subtitle_m3u8_url, display_id,
+            'Downloading m3u8 information',
+            'Failed to download m3u8 information',
+            fatal=True, data=None, headers={}, query={})
+        if res is not False:
+            subtitle_m3u8_doc, _ = res
+            for line in subtitle_m3u8_doc.split('\n'):
+                if 'EXT-X-MEDIA:TYPE=SUBTITLES' in line:
+                    parts = line.split(',')
+                    for part in parts:
+                        if 'LANGUAGE' in part:
+                            lang = part[part.index('=') + 2:-1]
+                        elif 'URI' in part:
+                            uri = part[part.index('=') + 2:-1]
+                    res = self._download_webpage_handle(
+                        uri, display_id,
+                        'Downloading m3u8 information',
+                        'Failed to download m3u8 information',
+                        fatal=True, data=None, headers={}, query={})
+                    doc, _ = res
+                    for l in doc.split('\n'):
+                        if not l.startswith('#'):
+                            subtitles[lang] = [{'url': uri[:-uri[::-1].index('/')] + l}]
+                            break
+
         return {
             'id': video_id,
             'display_id': display_id,
@@ -134,4 +163,5 @@ class RoosterTeethIE(InfoExtractor):
             'formats': formats,
             'channel_id': attributes.get('channel_id'),
             'duration': int_or_none(attributes.get('length')),
+            'subtitles': subtitles
         }

youtube_dlc/extractor/youtube.py

@@ -1817,6 +1817,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         if not isinstance(video_info, dict):
             video_info = {}
 
+        playable_in_embed = try_get(
+            player_response, lambda x: x['playabilityStatus']['playableInEmbed'])
+
         video_details = try_get(
             player_response, lambda x: x['videoDetails'], dict) or {}
 
@@ -2538,6 +2541,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             'release_date': release_date,
             'release_year': release_year,
             'subscriber_count': subscriber_count,
+            'playable_in_embed': playable_in_embed,
         }
 
 
@@ -3620,8 +3624,8 @@ class YoutubeSearchIE(SearchInfoExtractor, YoutubeBaseInfoExtractor):
                 description = try_get(video, lambda x: x['descriptionSnippet']['runs'][0]['text'], compat_str)
                 duration = parse_duration(try_get(video, lambda x: x['lengthText']['simpleText'], compat_str))
                 view_count_text = try_get(video, lambda x: x['viewCountText']['simpleText'], compat_str) or ''
-                view_count = int_or_none(self._search_regex(
-                    r'^(\d+)', re.sub(r'\s', '', view_count_text),
+                view_count = str_to_int(self._search_regex(
+                    r'^([\d,]+)', re.sub(r'\s', '', view_count_text),
                     'view count', default=None))
                 uploader = try_get(video, lambda x: x['ownerText']['runs'][0]['text'], compat_str)
                 total += 1

youtube_dlc/options.py

@@ -785,7 +785,7 @@ def parseOpts(overrideArguments=None):
     verbosity.add_option(
         '-C', '--call-home',
         dest='call_home', action='store_true', default=False,
-        help='Contact the youtube-dlc server for debugging')
+        help='[Broken] Contact the youtube-dlc server for debugging')
     verbosity.add_option(
         '--no-call-home',
         dest='call_home', action='store_false',
@@ -834,8 +834,16 @@ def parseOpts(overrideArguments=None):
         help=optparse.SUPPRESS_HELP)
     filesystem.add_option(
         '-w', '--no-overwrites',
-        action='store_true', dest='nooverwrites', default=False,
-        help='Do not overwrite files')
+        action='store_false', dest='overwrites', default=None,
+        help='Do not overwrite any files')
+    filesystem.add_option(
+        '--force-overwrites', '--yes-overwrites',
+        action='store_true', dest='overwrites',
+        help='Overwrite all video and metadata files. This option includes --no-continue')
+    filesystem.add_option(
+        '--no-force-overwrites',
+        action='store_const', dest='overwrites', const=None,
+        help='Do not overwrite the video, but overwrite related files (default)')
     filesystem.add_option(
         '-c', '--continue',
         action='store_true', dest='continue_dl', default=True,

youtube_dlc/postprocessor/common.py

@@ -37,7 +37,25 @@ class PostProcessor(object):
             self.PP_NAME = self.__class__.__name__[:-2]
 
     def to_screen(self, text, *args, **kwargs):
-        return self._downloader.to_screen('[%s] %s' % (self.PP_NAME, text), *args, **kwargs)
+        if self._downloader:
+            return self._downloader.to_screen('[%s] %s' % (self.PP_NAME, text), *args, **kwargs)
+
+    def report_warning(self, text, *args, **kwargs):
+        if self._downloader:
+            return self._downloader.report_warning(text, *args, **kwargs)
+
+    def report_error(self, text, *args, **kwargs):
+        if self._downloader:
+            return self._downloader.report_error(text, *args, **kwargs)
+
+    def write_debug(self, text, *args, **kwargs):
+        if self.get_param('verbose', False):
+            return self._downloader.to_screen('[debug] %s' % text, *args, **kwargs)
+
+    def get_param(self, name, default=None, *args, **kwargs):
+        if self._downloader:
+            return self._downloader.params.get(name, default, *args, **kwargs)
+        return default
 
     def set_downloader(self, downloader):
         """Sets the downloader for this PP."""
@@ -64,10 +82,10 @@ class PostProcessor(object):
         try:
             os.utime(encodeFilename(path), (atime, mtime))
         except Exception:
-            self._downloader.report_warning(errnote)
+            self.report_warning(errnote)
 
     def _configuration_args(self, default=[]):
-        args = self._downloader.params.get('postprocessor_args', {})
+        args = self.get_param('postprocessor_args', {})
         if isinstance(args, list):  # for backward compatibility
             args = {'default': args, 'sponskrub': []}
         return cli_configuration_args(args, self.PP_NAME.lower(), args.get('default', []))

youtube_dlc/postprocessor/embedthumbnail.py

@@ -41,8 +41,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
         thumbnail_filename = info['thumbnails'][-1]['filename']
 
         if not os.path.exists(encodeFilename(thumbnail_filename)):
-            self._downloader.report_warning(
-                'Skipping embedding the thumbnail because the file is missing.')
+            self.report_warning('Skipping embedding the thumbnail because the file is missing.')
             return [], info
 
         def is_webp(path):
@@ -125,8 +124,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
 
             self.to_screen('Adding thumbnail to "%s"' % filename)
 
-            if self._downloader.params.get('verbose', False):
-                self._downloader.to_screen('[debug] AtomicParsley command line: %s' % shell_quote(cmd))
+            self.verbose_message('AtomicParsley command line: %s' % shell_quote(cmd))
 
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             stdout, stderr = process_communicate_or_kill(p)
@@ -140,7 +138,7 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
             # for formats that don't support thumbnails (like 3gp) AtomicParsley
             # won't create to the temporary file
             if b'No changes' in stdout:
-                self._downloader.report_warning('The file format doesn\'t support embedding a thumbnail')
+                self.report_warning('The file format doesn\'t support embedding a thumbnail')
             else:
                 os.remove(encodeFilename(filename))
                 os.rename(encodeFilename(temp_filename), encodeFilename(filename))

youtube_dlc/postprocessor/ffmpeg.py

@@ -68,8 +68,7 @@ class FFmpegPostProcessor(PostProcessor):
                 self._versions[self.basename], required_version):
             warning = 'Your copy of %s is outdated, update %s to version %s or newer if you encounter any errors.' % (
                 self.basename, self.basename, required_version)
-            if self._downloader:
-                self._downloader.report_warning(warning)
+            self.report_warning(warning)
 
     @staticmethod
     def get_versions(downloader=None):
@@ -99,11 +98,11 @@ class FFmpegPostProcessor(PostProcessor):
         self._paths = None
         self._versions = None
         if self._downloader:
-            prefer_ffmpeg = self._downloader.params.get('prefer_ffmpeg', True)
-            location = self._downloader.params.get('ffmpeg_location')
+            prefer_ffmpeg = self.get_param('prefer_ffmpeg', True)
+            location = self.get_param('ffmpeg_location')
             if location is not None:
                 if not os.path.exists(location):
-                    self._downloader.report_warning(
+                    self.report_warning(
                         'ffmpeg-location %s does not exist! '
                         'Continuing without avconv/ffmpeg.' % (location))
                     self._versions = {}
@@ -111,7 +110,7 @@ class FFmpegPostProcessor(PostProcessor):
                 elif not os.path.isdir(location):
                     basename = os.path.splitext(os.path.basename(location))[0]
                     if basename not in programs:
-                        self._downloader.report_warning(
+                        self.report_warning(
                             'Cannot identify executable %s, its basename should be one of %s. '
                             'Continuing without avconv/ffmpeg.' %
                             (location, ', '.join(programs)))
@@ -177,9 +176,7 @@ class FFmpegPostProcessor(PostProcessor):
                     encodeFilename(self.executable, True),
                     encodeArgument('-i')]
             cmd.append(encodeFilename(self._ffmpeg_filename_argument(path), True))
-            if self._downloader.params.get('verbose', False):
-                self._downloader.to_screen(
-                    '[debug] %s command line: %s' % (self.basename, shell_quote(cmd)))
+            self.write_debug('%s command line: %s' % (self.basename, shell_quote(cmd)))
             handle = subprocess.Popen(
                 cmd, stderr=subprocess.PIPE,
                 stdout=subprocess.PIPE, stdin=subprocess.PIPE)
@@ -228,8 +225,7 @@ class FFmpegPostProcessor(PostProcessor):
                 + [encodeArgument(o) for o in opts]
                 + [encodeFilename(self._ffmpeg_filename_argument(out_path), True)])
 
-        if self._downloader.params.get('verbose', False):
-            self._downloader.to_screen('[debug] ffmpeg command line: %s' % shell_quote(cmd))
+        self.write_debug('ffmpeg command line: %s' % shell_quote(cmd))
         p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
         stdout, stderr = process_communicate_or_kill(p)
         if p.returncode != 0:
@@ -566,8 +562,7 @@ class FFmpegMergerPP(FFmpegPostProcessor):
                        'youtube-dlc will download single file media. '
                        'Update %s to version %s or newer to fix this.') % (
                            self.basename, self.basename, required_version)
-            if self._downloader:
-                self._downloader.report_warning(warning)
+            self.report_warning(warning)
             return False
         return True
 
@@ -656,7 +651,7 @@ class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):
             new_file = subtitles_filename(filename, lang, new_ext, info.get('ext'))
 
             if ext in ('dfxp', 'ttml', 'tt'):
-                self._downloader.report_warning(
+                self.report_warning(
                     'You have requested to convert dfxp (TTML) subtitles into another format, '
                     'which results in style information loss')
 

youtube_dlc/postprocessor/sponskrub.py

@@ -46,16 +46,16 @@ class SponSkrubPP(PostProcessor):
             self.to_screen('Skipping sponskrub since it is not a YouTube video')
             return [], information
         if self.cutout and not self.force and not information.get('__real_download', False):
-            self._downloader.to_screen(
-                '[sponskrub] Skipping sponskrub since the video was already downloaded. '
+            self.report_warning(
+                'Skipping sponskrub since the video was already downloaded. '
                 'Use --sponskrub-force to run sponskrub anyway')
             return [], information
 
         self.to_screen('Trying to %s sponsor sections' % ('remove' if self.cutout else 'mark'))
         if self.cutout:
-            self._downloader.to_screen('WARNING: Cutting out sponsor segments will cause the subtitles to go out of sync.')
+            self.report_warning('Cutting out sponsor segments will cause the subtitles to go out of sync.')
             if not information.get('__real_download', False):
-                self._downloader.to_screen('WARNING: If sponskrub is run multiple times, unintended parts of the video could be cut out.')
+                self.report_warning('If sponskrub is run multiple times, unintended parts of the video could be cut out.')
 
         filename = information['filepath']
         temp_filename = filename + '.' + self._temp_ext + os.path.splitext(filename)[1]
@@ -68,8 +68,7 @@ class SponSkrubPP(PostProcessor):
         cmd += ['--', information['id'], filename, temp_filename]
         cmd = [encodeArgument(i) for i in cmd]
 
-        if self._downloader.params.get('verbose', False):
-            self._downloader.to_screen('[debug] sponskrub command line: %s' % shell_quote(cmd))
+        self.write_debug('sponskrub command line: %s' % shell_quote(cmd))
         p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
         stdout, stderr = p.communicate()
 

youtube_dlc/postprocessor/xattrpp.py

@@ -57,16 +57,16 @@ class XAttrMetadataPP(PostProcessor):
             return [], info
 
         except XAttrUnavailableError as e:
-            self._downloader.report_error(str(e))
+            self.report_error(str(e))
             return [], info
 
         except XAttrMetadataError as e:
             if e.reason == 'NO_SPACE':
-                self._downloader.report_warning(
+                self.report_warning(
                     'There\'s no disk space left, disk quota exceeded or filesystem xattr limit exceeded. '
                     + (('Some ' if num_written else '') + 'extended attributes are not written.').capitalize())
             elif e.reason == 'VALUE_TOO_LONG':
-                self._downloader.report_warning(
+                self.report_warning(
                     'Unable to write extended attributes due to too long values.')
             else:
                 msg = 'This filesystem doesn\'t support extended attributes. '
@@ -74,5 +74,5 @@ class XAttrMetadataPP(PostProcessor):
                     msg += 'You need to use NTFS.'
                 else:
                     msg += '(You may have to enable them in your /etc/fstab)'
-                self._downloader.report_error(msg)
+                self.report_error(msg)
             return [], info

youtube_dlc/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.01.08'
+__version__ = '2021.01.10'