1
0
mirror of https://github.com/yt-dlp/yt-dlp.git synced 2026-01-12 09:51:15 +00:00

Compare commits

..

23 Commits

Author SHA1 Message Date
pukkandan
778763d9e6 Release 2022.08.18.36 2022-08-19 10:22:49 +05:30
pukkandan
6657e7834f Use compat.re 2022-08-19 09:49:52 +05:30
pukkandan
5dceb21df6 Revert "Remove Python 3.6 support"
This reverts commit 6929b41a21.
2022-08-19 09:49:29 +05:30
pukkandan
bf6bdf22de Revert "[ffmpeg] Set ffmpeg_location in a contextvar"
This reverts commit 6a7d3a0a09.
2022-08-19 09:49:26 +05:30
github-actions
b76e9cedb3 [version] update
Created by: pukkandan

:ci skip all :ci run dl
2022-08-19 00:11:11 +00:00
pukkandan
48c88e088c Release 2022.08.19 2022-08-19 05:08:22 +05:30
pukkandan
a831c2ea90 [cleanup] Misc 2022-08-19 05:08:21 +05:30
pukkandan
be13a6e525 [jsinterp] Bring on-par with youtube-dl
Code from: https://github.com/ytdl-org/youtube-dl/pull/31175, https://github.com/ytdl-org/youtube-dl/pull/31182

Authored by pukkandan, dirkf
2022-08-19 05:08:21 +05:30
bashonly
8a3da4c68c [extractor/instagram] Fix bugs in 7d3b98be4c (#4701)
Authored by: bashonly
2022-08-19 03:45:49 +05:30
nixxo
4d37d4a77c [extractor/rai] Minor fix (#4700)
Closes #4691, #4690
2022-08-19 02:28:59 +05:30
bashonly
7d3b98be4c [extractor/instagram] Fix extraction (#4696)
Closes #4657, #4532, #4475
Authored by: bashonly, pritam20ps05
2022-08-19 02:27:46 +05:30
Elyse
2b3e43e247 [extractor/rtbf] Fix stream extractor (#4671)
Closes #4656
Authored by: elyse0
2022-08-19 01:42:04 +05:30
Alexander Seiler
f60ef66371 [extractor/zattoo] Fix Zattoo resellers (#4675)
Closes #4630
Authored by: goggle
2022-08-19 01:27:51 +05:30
pukkandan
25836db6be [extractor/youtube] Add fallback to phantomjs
Related #4635
2022-08-18 21:35:18 +05:30
pukkandan
587021cd9f [phantomjs] Add function to execute JS without a DOM
Authored by: MinePlayersPE, pukkandan
2022-08-18 21:34:47 +05:30
pukkandan
580ce00782 [youtube] Improve signature caching
and refactor related functions
2022-08-18 21:33:30 +05:30
ChillingPepper
2f1a299c50 [extractor/SovietsCloset] Fix extractor (#4688)
Closes #4200 
Authored by: ChillingPepper
2022-08-18 16:44:45 +05:30
pukkandan
f6ca640b12 [jsinterp] Fix for youtube player 1f7d5369
Closes #4635 again
2022-08-18 16:38:35 +05:30
pukkandan
3ce2933693 [youtube] Fix error reporting of "Incomplete data"
Related: #4669
2022-08-16 22:01:48 +05:30
pukkandan
c200096c03 Fix bug in --download-archive
Closes #4668
2022-08-16 22:00:51 +05:30
pukkandan
6d3e7424bf [jsinterp] Fix for youtube player c81bbb4a 2022-08-16 06:53:45 +05:30
pukkandan
5c6d2ef9d1 [youtube] Improve format sorting for IOS formats
When no itag/resolution is available for reference, use the closest resolution
2022-08-15 14:04:05 +05:30
Lesmiscore
460eb9c50e [build] Exclude devscripts from installs
Closes #4667
2022-08-15 13:51:35 +05:30
39 changed files with 1251 additions and 386 deletions

View File

@@ -18,7 +18,7 @@ body:
options:
- label: I'm reporting a broken site
required: true
- label: I've verified that I'm running yt-dlp version **2022.08.14** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label: I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
required: true
- label: I've checked that all provided URLs are playable in a browser with the same IP and same login details
required: true
@@ -62,7 +62,7 @@ body:
[debug] Command-line config: ['-vU', 'test:youtube']
[debug] Portable config "yt-dlp.conf": ['-i']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [9d339c4] (win32_exe)
[debug] yt-dlp version 2022.08.19 [9d339c4] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
@@ -70,8 +70,8 @@ body:
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.14, Current version: 2022.08.14
yt-dlp is up to date (2022.08.14)
Latest version: 2022.08.19, Current version: 2022.08.19
yt-dlp is up to date (2022.08.19)
<more lines>
render: shell
validations:

View File

@@ -18,7 +18,7 @@ body:
options:
- label: I'm reporting a new site support request
required: true
- label: I've verified that I'm running yt-dlp version **2022.08.14** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label: I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
required: true
- label: I've checked that all provided URLs are playable in a browser with the same IP and same login details
required: true
@@ -74,7 +74,7 @@ body:
[debug] Command-line config: ['-vU', 'test:youtube']
[debug] Portable config "yt-dlp.conf": ['-i']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [9d339c4] (win32_exe)
[debug] yt-dlp version 2022.08.19 [9d339c4] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
@@ -82,8 +82,8 @@ body:
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.14, Current version: 2022.08.14
yt-dlp is up to date (2022.08.14)
Latest version: 2022.08.19, Current version: 2022.08.19
yt-dlp is up to date (2022.08.19)
<more lines>
render: shell
validations:

View File

@@ -18,7 +18,7 @@ body:
options:
- label: I'm requesting a site-specific feature
required: true
- label: I've verified that I'm running yt-dlp version **2022.08.14** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label: I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
required: true
- label: I've checked that all provided URLs are playable in a browser with the same IP and same login details
required: true
@@ -70,7 +70,7 @@ body:
[debug] Command-line config: ['-vU', 'test:youtube']
[debug] Portable config "yt-dlp.conf": ['-i']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [9d339c4] (win32_exe)
[debug] yt-dlp version 2022.08.19 [9d339c4] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
@@ -78,8 +78,8 @@ body:
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.14, Current version: 2022.08.14
yt-dlp is up to date (2022.08.14)
Latest version: 2022.08.19, Current version: 2022.08.19
yt-dlp is up to date (2022.08.19)
<more lines>
render: shell
validations:

View File

@@ -18,7 +18,7 @@ body:
options:
- label: I'm reporting a bug unrelated to a specific site
required: true
- label: I've verified that I'm running yt-dlp version **2022.08.14** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label: I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
required: true
- label: I've checked that all provided URLs are playable in a browser with the same IP and same login details
required: true
@@ -55,7 +55,7 @@ body:
[debug] Command-line config: ['-vU', 'test:youtube']
[debug] Portable config "yt-dlp.conf": ['-i']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [9d339c4] (win32_exe)
[debug] yt-dlp version 2022.08.19 [9d339c4] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
@@ -63,8 +63,8 @@ body:
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.14, Current version: 2022.08.14
yt-dlp is up to date (2022.08.14)
Latest version: 2022.08.19, Current version: 2022.08.19
yt-dlp is up to date (2022.08.19)
<more lines>
render: shell
validations:

View File

@@ -20,7 +20,7 @@ body:
required: true
- label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required: true
- label: I've verified that I'm running yt-dlp version **2022.08.14** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label: I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
required: true
- label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required: true
@@ -51,7 +51,7 @@ body:
[debug] Command-line config: ['-vU', 'test:youtube']
[debug] Portable config "yt-dlp.conf": ['-i']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [9d339c4] (win32_exe)
[debug] yt-dlp version 2022.08.19 [9d339c4] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
@@ -59,7 +59,7 @@ body:
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.14, Current version: 2022.08.14
yt-dlp is up to date (2022.08.14)
Latest version: 2022.08.19, Current version: 2022.08.19
yt-dlp is up to date (2022.08.19)
<more lines>
render: shell

View File

@@ -26,7 +26,7 @@ body:
required: true
- label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required: true
- label: I've verified that I'm running yt-dlp version **2022.08.14** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label: I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
required: true
- label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
required: true
@@ -57,7 +57,7 @@ body:
[debug] Command-line config: ['-vU', 'test:youtube']
[debug] Portable config "yt-dlp.conf": ['-i']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [9d339c4] (win32_exe)
[debug] yt-dlp version 2022.08.19 [9d339c4] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
@@ -65,7 +65,7 @@ body:
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.14, Current version: 2022.08.14
yt-dlp is up to date (2022.08.14)
Latest version: 2022.08.19, Current version: 2022.08.19
yt-dlp is up to date (2022.08.19)
<more lines>
render: shell

View File

@@ -264,7 +264,7 @@ jobs:
- name: Make Update spec
run: |
echo "# This file is used for regulating self-update" >> _update_spec
echo "lock 2022.07.18 .+ Python 3.6" >> _update_spec
echo "lock 2022.08.18.36 .+ Python 3.6" >> _update_spec
- name: Make SHA2-SUMS files
run: |
sha256sum artifact/yt-dlp | awk '{print $1 " yt-dlp"}' >> SHA2-256SUMS

View File

@@ -10,7 +10,7 @@ jobs:
matrix:
os: [ubuntu-latest]
# CPython 3.9 is in quick-test
python-version: ['3.7', '3.10', 3.11-dev, pypy-3.7, pypy-3.8]
python-version: ['3.6', '3.7', '3.10', 3.11-dev, pypy-3.6, pypy-3.7, pypy-3.8]
run-tests-ext: [sh]
include:
# atleast one of each CPython/PyPy tests must be in windows

View File

@@ -25,7 +25,7 @@ jobs:
fail-fast: true
matrix:
os: [ubuntu-latest]
python-version: ['3.7', '3.10', 3.11-dev, pypy-3.7, pypy-3.8]
python-version: ['3.6', '3.7', '3.10', 3.11-dev, pypy-3.6, pypy-3.7, pypy-3.8]
run-tests-ext: [sh]
include:
# atleast one of each CPython/PyPy tests must be in windows

View File

@@ -11,6 +11,29 @@
-->
### 2022.08.18.36
* Revert "[ffmpeg] Set `ffmpeg_location` in a contextvar"
* Revert "Remove Python 3.6 support"
* Use `compat.re`
### 2022.08.19
* Fix bug in `--download-archive`
* [jsinterp] **Fix for new youtube players** and related improvements by [dirkf](https://github.com/dirkf), [pukkandan](https://github.com/pukkandan)
* [phantomjs] Add function to execute JS without a DOM by [MinePlayersPE](https://github.com/MinePlayersPE), [pukkandan](https://github.com/pukkandan)
* [build] Exclude devscripts from installs by [Lesmiscore](https://github.com/Lesmiscore)
* [cleanup] Misc fixes and cleanup
* [extractor/youtube] **Add fallback to phantomjs** for nsig
* [extractor/youtube] Fix error reporting of "Incomplete data"
* [extractor/youtube] Improve format sorting for IOS formats
* [extractor/youtube] Improve signature caching
* [extractor/instagram] Fix extraction by [bashonly](https://github.com/bashonly), [pritam20ps05](https://github.com/pritam20ps05)
* [extractor/rai] Minor fix by [nixxo](https://github.com/nixxo)
* [extractor/rtbf] Fix stream extractor by [elyse0](https://github.com/elyse0)
* [extractor/SovietsCloset] Fix extractor by [ChillingPepper](https://github.com/ChillingPepper)
* [extractor/zattoo] Fix Zattoo resellers by [goggle](https://github.com/goggle)
### 2022.08.14
* Merge youtube-dl: Upto [commit/d231b56](https://github.com/ytdl-org/youtube-dl/commit/d231b56)
@@ -19,8 +42,7 @@
* [extractor] Fix format sorting of `channels`
* [ffmpeg] Disable avconv unless `--prefer-avconv`
* [ffmpeg] Smarter detection of ffprobe filename
* [patreon] Ignore erroneous media attachments by [coletdjnz](https://github.com/coletdjnz)
* [postprocessor/embedthumbnail] Detect `libatomicparsley.so`
* [embedthumbnail] Detect `libatomicparsley.so`
* [ThumbnailsConvertor] Fix conversion after `fixup_webp`
* [utils] Fix `get_compatible_ext`
* [build] Fix changelog
@@ -30,6 +52,7 @@
* [cleanup] Misc fixes and cleanup
* [extractor/moview] Add extractor by [HobbyistDev](https://github.com/HobbyistDev)
* [extractor/parler] Add extractor by [palewire](https://github.com/palewire)
* [extractor/patreon] Ignore erroneous media attachments by [coletdjnz](https://github.com/coletdjnz)
* [extractor/truth] Add extractor by [palewire](https://github.com/palewire)
* [extractor/aenetworks] Add formats parameter by [jacobtruman](https://github.com/jacobtruman)
* [extractor/crunchyroll] Improve `_VALID_URL`s

View File

@@ -71,7 +71,7 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
# NEW FEATURES
* Merged with **youtube-dl v2021.12.17+ [commit/d231b56](https://github.com/ytdl-org/youtube-dl/commit/d231b56717c73ee597d2e077d11b69ed48a1b02d)**<!--([exceptions](https://github.com/yt-dlp/yt-dlp/issues/21))--> and **youtube-dlc v2020.11.11-3+ [commit/f9401f2](https://github.com/blackjack4494/yt-dlc/commit/f9401f2a91987068139c5f757b12fc711d4c0cee)**: You get all the features and patches of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) in addition to the latest [youtube-dl](https://github.com/ytdl-org/youtube-dl)
* Merged with **youtube-dl v2021.12.17+ [commit/b0a60ce](https://github.com/ytdl-org/youtube-dl/commit/b0a60ce2032172aeaaf27fe3866ab72768f10cb2)**<!--([exceptions](https://github.com/yt-dlp/yt-dlp/issues/21))--> and **youtube-dlc v2020.11.11-3+ [commit/f9401f2](https://github.com/blackjack4494/yt-dlc/commit/f9401f2a91987068139c5f757b12fc711d4c0cee)**: You get all the features and patches of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) in addition to the latest [youtube-dl](https://github.com/ytdl-org/youtube-dl)
* **[SponsorBlock Integration](#sponsorblock-options)**: You can mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
@@ -329,7 +329,7 @@ You will need the build tools `python` (3.6+), `zip`, `make` (GNU), `pandoc`\* a
After installing these, simply run `make`.
You can also run `make yt-dlp` instead to compile only the binary without updating any of the additional files. (The dependencies marked with **\*** are not needed for this)
You can also run `make yt-dlp` instead to compile only the binary without updating any of the additional files. (The build tools marked with **\*** are not needed for this)
### Standalone Py2Exe Builds (Windows)

View File

@@ -16,6 +16,7 @@ from devscripts.utils import read_version, write_file
def get_new_version(revision):
return read_version()
version = datetime.utcnow().strftime('%Y.%m.%d')
if revision:

View File

@@ -81,7 +81,7 @@ def version_to_list(version):
def dependency_options():
# Due to the current implementation, these are auto-detected, but explicitly add them just in case
dependencies = [pycryptodome_module(), 'mutagen', 'brotli', 'certifi', 'websockets']
excluded_modules = ['test', 'ytdlp_plugins', 'youtube_dl', 'youtube_dlc']
excluded_modules = ('youtube_dl', 'youtube_dlc', 'test', 'ytdlp_plugins', 'devscripts')
yield from (f'--hidden-import={module}' for module in dependencies)
yield '--collect-submodules=websockets'

View File

@@ -28,7 +28,7 @@ REQUIREMENTS = read_file('requirements.txt').splitlines()
def packages():
if setuptools_available:
return find_packages(exclude=('youtube_dl', 'youtube_dlc', 'test', 'ytdlp_plugins'))
return find_packages(exclude=('youtube_dl', 'youtube_dlc', 'test', 'ytdlp_plugins', 'devscripts'))
return [
'yt_dlp', 'yt_dlp.extractor', 'yt_dlp.downloader', 'yt_dlp.postprocessor', 'yt_dlp.compat',
@@ -126,7 +126,7 @@ setup(
url='https://github.com/yt-dlp/yt-dlp',
packages=packages(),
install_requires=REQUIREMENTS,
python_requires='>=3.7',
python_requires='>=3.6',
project_urls={
'Documentation': 'https://github.com/yt-dlp/yt-dlp#readme',
'Source': 'https://github.com/yt-dlp/yt-dlp',
@@ -138,6 +138,7 @@ setup(
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
'Programming Language :: Python',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',

View File

@@ -128,6 +128,8 @@
- **bbc.co.uk:iplayer:group**
- **bbc.co.uk:playlist**
- **BBVTV**: [<abbr title="netrc machine"><em>bbvtv</em></abbr>]
- **BBVTVLive**: [<abbr title="netrc machine"><em>bbvtv</em></abbr>]
- **BBVTVRecordings**: [<abbr title="netrc machine"><em>bbvtv</em></abbr>]
- **Beatport**
- **Beeg**
- **BehindKink**
@@ -348,6 +350,8 @@
- **ehftv**
- **eHow**
- **EinsUndEinsTV**: [<abbr title="netrc machine"><em>1und1tv</em></abbr>]
- **EinsUndEinsTVLive**: [<abbr title="netrc machine"><em>1und1tv</em></abbr>]
- **EinsUndEinsTVRecordings**: [<abbr title="netrc machine"><em>1und1tv</em></abbr>]
- **Einthusan**
- **eitb.tv**
- **EllenTube**
@@ -375,6 +379,8 @@
- **EuropeanTour**
- **EUScreen**
- **EWETV**: [<abbr title="netrc machine"><em>ewetv</em></abbr>]
- **EWETVLive**: [<abbr title="netrc machine"><em>ewetv</em></abbr>]
- **EWETVRecordings**: [<abbr title="netrc machine"><em>ewetv</em></abbr>]
- **ExpoTV**
- **Expressen**
- **ExtremeTube**
@@ -454,6 +460,8 @@
- **GiantBomb**
- **Giga**
- **GlattvisionTV**: [<abbr title="netrc machine"><em>glattvisiontv</em></abbr>]
- **GlattvisionTVLive**: [<abbr title="netrc machine"><em>glattvisiontv</em></abbr>]
- **GlattvisionTVRecordings**: [<abbr title="netrc machine"><em>glattvisiontv</em></abbr>]
- **Glide**: Glide mobile video messages (glide.me)
- **Globo**: [<abbr title="netrc machine"><em>globo</em></abbr>]
- **GloboArticle**
@@ -715,6 +723,8 @@
- **MLSSoccer**
- **Mnet**
- **MNetTV**: [<abbr title="netrc machine"><em>mnettv</em></abbr>]
- **MNetTVLive**: [<abbr title="netrc machine"><em>mnettv</em></abbr>]
- **MNetTVRecordings**: [<abbr title="netrc machine"><em>mnettv</em></abbr>]
- **MochaVideo**
- **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
- **Mofosex**
@@ -801,7 +811,9 @@
- **netease:program**: 网易云音乐 - 电台节目
- **netease:singer**: 网易云音乐 - 歌手
- **netease:song**: 网易云音乐
- **NetPlus**: [<abbr title="netrc machine"><em>netplus</em></abbr>]
- **NetPlusTV**: [<abbr title="netrc machine"><em>netplus</em></abbr>]
- **NetPlusTVLive**: [<abbr title="netrc machine"><em>netplus</em></abbr>]
- **NetPlusTVRecordings**: [<abbr title="netrc machine"><em>netplus</em></abbr>]
- **Netverse**
- **NetversePlaylist**
- **Netzkino**
@@ -906,6 +918,8 @@
- **orf:radio**
- **orf:tvthek**: ORF TVthek
- **OsnatelTV**: [<abbr title="netrc machine"><em>osnateltv</em></abbr>]
- **OsnatelTVLive**: [<abbr title="netrc machine"><em>osnateltv</em></abbr>]
- **OsnatelTVRecordings**: [<abbr title="netrc machine"><em>osnateltv</em></abbr>]
- **OutsideTV**
- **PacktPub**: [<abbr title="netrc machine"><em>packtpub</em></abbr>]
- **PacktPubCourse**
@@ -1013,6 +1027,8 @@
- **qqmusic:singer**: QQ音乐 - 歌手
- **qqmusic:toplist**: QQ音乐 - 排行榜
- **QuantumTV**: [<abbr title="netrc machine"><em>quantumtv</em></abbr>]
- **QuantumTVLive**: [<abbr title="netrc machine"><em>quantumtv</em></abbr>]
- **QuantumTVRecordings**: [<abbr title="netrc machine"><em>quantumtv</em></abbr>]
- **Qub**
- **R7**
- **R7Article**
@@ -1121,7 +1137,11 @@
- **safari:course**: [<abbr title="netrc machine"><em>safari</em></abbr>] safaribooksonline.com online courses
- **Saitosan**
- **SAKTV**: [<abbr title="netrc machine"><em>saktv</em></abbr>]
- **SAKTVLive**: [<abbr title="netrc machine"><em>saktv</em></abbr>]
- **SAKTVRecordings**: [<abbr title="netrc machine"><em>saktv</em></abbr>]
- **SaltTV**: [<abbr title="netrc machine"><em>salttv</em></abbr>]
- **SaltTVLive**: [<abbr title="netrc machine"><em>salttv</em></abbr>]
- **SaltTVRecordings**: [<abbr title="netrc machine"><em>salttv</em></abbr>]
- **SampleFocus**
- **Sapo**: SAPO Vídeos
- **savefrom.net**
@@ -1494,6 +1514,8 @@
- **VShare**
- **VTM**
- **VTXTV**: [<abbr title="netrc machine"><em>vtxtv</em></abbr>]
- **VTXTVLive**: [<abbr title="netrc machine"><em>vtxtv</em></abbr>]
- **VTXTVRecordings**: [<abbr title="netrc machine"><em>vtxtv</em></abbr>]
- **VuClip**
- **Vupload**
- **VVVVID**
@@ -1503,6 +1525,8 @@
- **Wakanim**
- **Walla**
- **WalyTV**: [<abbr title="netrc machine"><em>walytv</em></abbr>]
- **WalyTVLive**: [<abbr title="netrc machine"><em>walytv</em></abbr>]
- **WalyTVRecordings**: [<abbr title="netrc machine"><em>walytv</em></abbr>]
- **wasdtv:clip**
- **wasdtv:record**
- **wasdtv:stream**

View File

@@ -28,8 +28,7 @@ class TestCompat(unittest.TestCase):
with self.assertWarns(DeprecationWarning):
compat.WINDOWS_VT_MODE
# TODO: Test submodule
# compat.asyncio.events # Must not raise error
compat.asyncio.events # Must not raise error
def test_compat_expanduser(self):
old_home = os.environ.get('HOME')

View File

@@ -7,8 +7,10 @@ import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import math
from yt_dlp.jsinterp import JSInterpreter
from yt_dlp.compat import re
from yt_dlp.jsinterp import JS_Undefined, JSInterpreter
class TestJSInterpreter(unittest.TestCase):
@@ -66,6 +68,9 @@ class TestJSInterpreter(unittest.TestCase):
jsi = JSInterpreter('function f(){return 0 && 1 || 2;}')
self.assertEqual(jsi.call_function('f'), 2)
jsi = JSInterpreter('function f(){return 0 ?? 42;}')
self.assertEqual(jsi.call_function('f'), 0)
def test_array_access(self):
jsi = JSInterpreter('function f(){var x = [1,2,3]; x[0] = 4; x[0] = 5; x[2.0] = 7; return x;}')
self.assertEqual(jsi.call_function('f'), [5, 2, 7])
@@ -212,6 +217,11 @@ class TestJSInterpreter(unittest.TestCase):
''')
self.assertEqual(jsi.call_function('x'), 7)
jsi = JSInterpreter('''
function x() { return (l=[0,1,2,3], function(a, b){return a+b})((l[1], l[2]), l[3]) }
''')
self.assertEqual(jsi.call_function('x'), 5)
def test_void(self):
jsi = JSInterpreter('''
function x() { return void 42; }
@@ -224,6 +234,119 @@ class TestJSInterpreter(unittest.TestCase):
''')
self.assertEqual(jsi.call_function('x')([]), 1)
def test_null(self):
jsi = JSInterpreter('''
function x() { return null; }
''')
self.assertEqual(jsi.call_function('x'), None)
jsi = JSInterpreter('''
function x() { return [null > 0, null < 0, null == 0, null === 0]; }
''')
self.assertEqual(jsi.call_function('x'), [False, False, False, False])
jsi = JSInterpreter('''
function x() { return [null >= 0, null <= 0]; }
''')
self.assertEqual(jsi.call_function('x'), [True, True])
def test_undefined(self):
jsi = JSInterpreter('''
function x() { return undefined === undefined; }
''')
self.assertEqual(jsi.call_function('x'), True)
jsi = JSInterpreter('''
function x() { return undefined; }
''')
self.assertEqual(jsi.call_function('x'), JS_Undefined)
jsi = JSInterpreter('''
function x() { let v; return v; }
''')
self.assertEqual(jsi.call_function('x'), JS_Undefined)
jsi = JSInterpreter('''
function x() { return [undefined === undefined, undefined == undefined, undefined < undefined, undefined > undefined]; }
''')
self.assertEqual(jsi.call_function('x'), [True, True, False, False])
jsi = JSInterpreter('''
function x() { return [undefined === 0, undefined == 0, undefined < 0, undefined > 0]; }
''')
self.assertEqual(jsi.call_function('x'), [False, False, False, False])
jsi = JSInterpreter('''
function x() { return [undefined >= 0, undefined <= 0]; }
''')
self.assertEqual(jsi.call_function('x'), [False, False])
jsi = JSInterpreter('''
function x() { return [undefined > null, undefined < null, undefined == null, undefined === null]; }
''')
self.assertEqual(jsi.call_function('x'), [False, False, True, False])
jsi = JSInterpreter('''
function x() { return [undefined === null, undefined == null, undefined < null, undefined > null]; }
''')
self.assertEqual(jsi.call_function('x'), [False, True, False, False])
jsi = JSInterpreter('''
function x() { let v; return [42+v, v+42, v**42, 42**v, 0**v]; }
''')
for y in jsi.call_function('x'):
self.assertTrue(math.isnan(y))
jsi = JSInterpreter('''
function x() { let v; return v**0; }
''')
self.assertEqual(jsi.call_function('x'), 1)
jsi = JSInterpreter('''
function x() { let v; return [v>42, v<=42, v&&42, 42&&v]; }
''')
self.assertEqual(jsi.call_function('x'), [False, False, JS_Undefined, JS_Undefined])
jsi = JSInterpreter('function x(){return undefined ?? 42; }')
self.assertEqual(jsi.call_function('x'), 42)
def test_object(self):
jsi = JSInterpreter('''
function x() { return {}; }
''')
self.assertEqual(jsi.call_function('x'), {})
jsi = JSInterpreter('''
function x() { let a = {m1: 42, m2: 0 }; return [a["m1"], a.m2]; }
''')
self.assertEqual(jsi.call_function('x'), [42, 0])
jsi = JSInterpreter('''
function x() { let a; return a?.qq; }
''')
self.assertEqual(jsi.call_function('x'), JS_Undefined)
jsi = JSInterpreter('''
function x() { let a = {m1: 42, m2: 0 }; return a?.qq; }
''')
self.assertEqual(jsi.call_function('x'), JS_Undefined)
def test_regex(self):
jsi = JSInterpreter('''
function x() { let a=/,,[/,913,/](,)}/; }
''')
self.assertEqual(jsi.call_function('x'), None)
jsi = JSInterpreter('''
function x() { let a=/,,[/,913,/](,)}/; return a; }
''')
self.assertIsInstance(jsi.call_function('x'), re.Pattern)
jsi = JSInterpreter('''
function x() { let a=/,,[/,913,/](,)}/i; return a; }
''')
self.assertEqual(jsi.call_function('x').flags & re.I, re.I)
if __name__ == '__main__':
unittest.main()

View File

@@ -102,6 +102,14 @@ _NSIG_TESTS = [
'https://www.youtube.com/s/player/4c3f79c5/player_ias.vflset/en_US/base.js',
'TDCstCG66tEAO5pR9o', 'dbxNtZ14c-yWyw',
),
(
'https://www.youtube.com/s/player/c81bbb4a/player_ias.vflset/en_US/base.js',
'gre3EcLurNY2vqp94', 'Z9DfGxWP115WTg',
),
(
'https://www.youtube.com/s/player/1f7d5369/player_ias.vflset/en_US/base.js',
'batNX7sYqIJdkJ', 'IhOkL_zxbkOZBw',
),
]

View File

@@ -444,6 +444,7 @@ class YoutubeDL:
* index: Section number (Optional)
force_keyframes_at_cuts: Re-encode the video when downloading ranges to get precise cuts
noprogress: Do not print the progress bar
live_from_start: Whether to download livestreams videos from the start
The following parameters are not used by YoutubeDL itself, they are used by
the downloader (see yt_dlp/downloader/common.py):
@@ -590,8 +591,7 @@ class YoutubeDL:
for type_, stream in self._out_files.items_ if type_ != 'console'
})
# The code is left like this to be reused for future deprecations
MIN_SUPPORTED, MIN_RECOMMENDED = (3, 7), (3, 7)
MIN_SUPPORTED, MIN_RECOMMENDED = (3, 6), (3, 7)
current_version = sys.version_info[:2]
if current_version < MIN_RECOMMENDED:
msg = ('Support for Python version %d.%d has been deprecated. '
@@ -3443,7 +3443,7 @@ class YoutubeDL:
return False
vid_ids = [self._make_archive_id(info_dict)]
vid_ids.extend(info_dict.get('_old_archive_ids', []))
vid_ids.extend(info_dict.get('_old_archive_ids') or [])
return any(id_ in self.archive for id_ in vid_ids)
def record_download_archive(self, info_dict):

View File

@@ -1,8 +1,4 @@
try:
import contextvars # noqa: F401
except Exception:
raise Exception(
f'You are using an unsupported version of Python. Only Python versions 3.7 and above are supported by yt-dlp') # noqa: F541
f'You are using an unsupported version of Python. Only Python versions 3.6 and above are supported by yt-dlp' # noqa: F541
__license__ = 'Public Domain'
@@ -25,7 +21,6 @@ from .options import parseOpts
from .postprocessor import (
FFmpegExtractAudioPP,
FFmpegMergerPP,
FFmpegPostProcessor,
FFmpegSubtitlesConvertorPP,
FFmpegThumbnailsConvertorPP,
FFmpegVideoConvertorPP,
@@ -907,11 +902,6 @@ def _real_main(argv=None):
if print_extractor_information(opts, all_urls):
return
# We may need ffmpeg_location without having access to the YoutubeDL instance
# See https://github.com/yt-dlp/yt-dlp/issues/2191
if opts.ffmpeg_location:
FFmpegPostProcessor._ffmpeg_location.set(opts.ffmpeg_location)
with YoutubeDL(ydl_opts) as ydl:
pre_process = opts.update_self or opts.rm_cachedir
actual_use = all_urls or opts.load_info_filename

View File

@@ -3,6 +3,7 @@ import sys
import warnings
import xml.etree.ElementTree as etree
from . import re
from ._deprecated import * # noqa: F401, F403
from .compat_utils import passthrough_module
@@ -32,7 +33,6 @@ compat_os_name = os._name if os.name == 'java' else os.name
if compat_os_name == 'nt':
def compat_shlex_quote(s):
import re
return s if re.match(r'^[-_\w./]+$', s) else '"%s"' % s.replace('"', '\\"')
else:
from shlex import quote as compat_shlex_quote # noqa: F401

View File

@@ -22,14 +22,10 @@ import urllib.request
import xml.etree.ElementTree as etree
from subprocess import DEVNULL
# isort: split
import asyncio # noqa: F401
import re # noqa: F401
from asyncio import run as compat_asyncio_run # noqa: F401
from re import Pattern as compat_Pattern # noqa: F401
from re import match as compat_Match # noqa: F401
from .compat_utils import passthrough_module
from .compat_utils import passthrough_module # isort: split
from .asyncio import run as compat_asyncio_run # noqa: F401
from .re import Pattern as compat_Pattern # noqa: F401
from .re import match as compat_Match # noqa: F401
from ..dependencies import Cryptodome_AES as compat_pycrypto_AES # noqa: F401
from ..dependencies import brotli as compat_brotli # noqa: F401
from ..dependencies import websockets as compat_websockets # noqa: F401

23
yt_dlp/compat/asyncio.py Normal file
View File

@@ -0,0 +1,23 @@
# flake8: noqa: F405
from asyncio import * # noqa: F403
from .compat_utils import passthrough_module
passthrough_module(__name__, 'asyncio')
del passthrough_module
try:
run # >= 3.7
except NameError:
def run(coro):
try:
loop = get_event_loop()
except RuntimeError:
loop = new_event_loop()
set_event_loop(loop)
loop.run_until_complete(coro)
try:
all_tasks # >= 3.7
except NameError:
all_tasks = Task.all_tasks

18
yt_dlp/compat/re.py Normal file
View File

@@ -0,0 +1,18 @@
# flake8: noqa: F405
from re import * # F403
from .compat_utils import passthrough_module
passthrough_module(__name__, 're')
del passthrough_module
try:
Pattern # >= 3.7
except NameError:
Pattern = type(compile(''))
try:
Match # >= 3.7
except NameError:
Match = type(compile('').match(''))

View File

@@ -1,4 +1,3 @@
import asyncio
import contextlib
import os
import signal
@@ -6,6 +5,7 @@ import threading
from .common import FileDownloader
from .external import FFmpegFD
from ..compat import asyncio
from ..dependencies import websockets

View File

@@ -2200,17 +2200,41 @@ from .youtube import (
from .zapiks import ZapiksIE
from .zattoo import (
BBVTVIE,
BBVTVLiveIE,
BBVTVRecordingsIE,
EinsUndEinsTVIE,
EinsUndEinsTVLiveIE,
EinsUndEinsTVRecordingsIE,
EWETVIE,
EWETVLiveIE,
EWETVRecordingsIE,
GlattvisionTVIE,
GlattvisionTVLiveIE,
GlattvisionTVRecordingsIE,
MNetTVIE,
NetPlusIE,
MNetTVLiveIE,
MNetTVRecordingsIE,
NetPlusTVIE,
NetPlusTVLiveIE,
NetPlusTVRecordingsIE,
OsnatelTVIE,
OsnatelTVLiveIE,
OsnatelTVRecordingsIE,
QuantumTVIE,
QuantumTVLiveIE,
QuantumTVRecordingsIE,
SaltTVIE,
SaltTVLiveIE,
SaltTVRecordingsIE,
SAKTVIE,
SAKTVLiveIE,
SAKTVRecordingsIE,
VTXTVIE,
VTXTVLiveIE,
VTXTVRecordingsIE,
WalyTVIE,
WalyTVLiveIE,
WalyTVRecordingsIE,
ZattooIE,
ZattooLiveIE,
ZattooMoviesIE,

View File

@@ -11,7 +11,6 @@ import math
import netrc
import os
import random
import re
import sys
import time
import types
@@ -19,7 +18,7 @@ import urllib.parse
import urllib.request
import xml.etree.ElementTree
from ..compat import functools # isort: split
from ..compat import functools, re # isort: split
from ..compat import compat_etree_fromstring, compat_expanduser, compat_os_name
from ..downloader import FileDownloader
from ..downloader.f4m import get_base_url, remove_encrypted_media

View File

@@ -39,37 +39,42 @@ class InstagramBaseIE(InfoExtractor):
_NETRC_MACHINE = 'instagram'
_IS_LOGGED_IN = False
_API_BASE_URL = 'https://i.instagram.com/api/v1'
_LOGIN_URL = 'https://www.instagram.com/accounts/login'
_API_HEADERS = {
'X-IG-App-ID': '936619743392459',
'X-ASBD-ID': '198387',
'X-IG-WWW-Claim': '0',
'Origin': 'https://www.instagram.com',
'Accept': '*/*',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36',
}
def _perform_login(self, username, password):
if self._IS_LOGGED_IN:
return
login_webpage = self._download_webpage(
'https://www.instagram.com/accounts/login/', None,
note='Downloading login webpage', errnote='Failed to download login webpage')
self._LOGIN_URL, None, note='Downloading login webpage', errnote='Failed to download login webpage')
shared_data = self._parse_json(
self._search_regex(
r'window\._sharedData\s*=\s*({.+?});',
login_webpage, 'shared data', default='{}'),
None)
shared_data = self._parse_json(self._search_regex(
r'window\._sharedData\s*=\s*({.+?});', login_webpage, 'shared data', default='{}'), None)
login = self._download_json('https://www.instagram.com/accounts/login/ajax/', None, note='Logging in', headers={
'Accept': '*/*',
'X-IG-App-ID': '936619743392459',
'X-ASBD-ID': '198387',
'X-IG-WWW-Claim': '0',
'X-Requested-With': 'XMLHttpRequest',
'X-CSRFToken': shared_data['config']['csrf_token'],
'X-Instagram-AJAX': shared_data['rollout_hash'],
'Referer': 'https://www.instagram.com/',
}, data=urlencode_postdata({
'enc_password': f'#PWD_INSTAGRAM_BROWSER:0:{int(time.time())}:{password}',
'username': username,
'queryParams': '{}',
'optIntoOneTap': 'false',
'stopDeletionNonce': '',
'trustedDeviceRecords': '{}',
}))
login = self._download_json(
f'{self._LOGIN_URL}/ajax/', None, note='Logging in', headers={
**self._API_HEADERS,
'X-Requested-With': 'XMLHttpRequest',
'X-CSRFToken': shared_data['config']['csrf_token'],
'X-Instagram-AJAX': shared_data['rollout_hash'],
'Referer': 'https://www.instagram.com/',
}, data=urlencode_postdata({
'enc_password': f'#PWD_INSTAGRAM_BROWSER:0:{int(time.time())}:{password}',
'username': username,
'queryParams': '{}',
'optIntoOneTap': 'false',
'stopDeletionNonce': '',
'trustedDeviceRecords': '{}',
}))
if not login.get('authenticated'):
if login.get('message'):
@@ -134,7 +139,7 @@ class InstagramBaseIE(InfoExtractor):
}
def _extract_product_media(self, product_media):
media_id = product_media.get('code') or product_media.get('id')
media_id = product_media.get('code') or _pk_to_id(product_media.get('pk'))
vcodec = product_media.get('video_codec')
dash_manifest_raw = product_media.get('video_dash_manifest')
videos_list = product_media.get('video_versions')
@@ -179,7 +184,7 @@ class InstagramBaseIE(InfoExtractor):
user_info = product_info.get('user') or {}
info_dict = {
'id': product_info.get('code') or product_info.get('id'),
'id': product_info.get('code') or _pk_to_id(product_info.get('pk')),
'title': product_info.get('title') or f'Video by {user_info.get("username")}',
'description': traverse_obj(product_info, ('caption', 'text'), expected_type=str_or_none),
'timestamp': int_or_none(product_info.get('taken_at')),
@@ -360,49 +365,74 @@ class InstagramIE(InstagramBaseIE):
def _real_extract(self, url):
video_id, url = self._match_valid_url(url).group('id', 'url')
general_info = self._download_json(
f'https://www.instagram.com/graphql/query/?query_hash=9f8827793ef34641b2fb195d4d41151c'
f'&variables=%7B"shortcode":"{video_id}",'
'"parent_comment_count":10,"has_threaded_comments":true}', video_id, fatal=False, errnote=False,
headers={
'Accept': '*',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
'Authority': 'www.instagram.com',
'Referer': 'https://www.instagram.com',
'x-ig-app-id': '936619743392459',
})
media = traverse_obj(general_info, ('data', 'shortcode_media')) or {}
media, webpage = {}, ''
api_check = self._download_json(
f'{self._API_BASE_URL}/web/get_ruling_for_content/?content_type=MEDIA&target_id={_id_to_pk(video_id)}',
video_id, headers=self._API_HEADERS, fatal=False, note='Setting up session', errnote=False) or {}
csrf_token = self._get_cookies('https://www.instagram.com').get('csrftoken')
if not csrf_token:
self.report_warning('No csrf token set by Instagram API', video_id)
elif api_check.get('status') != 'ok':
self.report_warning('Instagram API is not granting access', video_id)
else:
if self._get_cookies(url).get('sessionid'):
media.update(traverse_obj(self._download_json(
f'{self._API_BASE_URL}/media/{_id_to_pk(video_id)}/info/', video_id,
fatal=False, note='Downloading video info', headers={
**self._API_HEADERS,
'X-CSRFToken': csrf_token.value,
}), ('items', 0)) or {})
if media:
return self._extract_product(media)
variables = {
'shortcode': video_id,
'child_comment_count': 3,
'fetch_comment_count': 40,
'parent_comment_count': 24,
'has_threaded_comments': True,
}
general_info = self._download_json(
'https://www.instagram.com/graphql/query/', video_id, fatal=False,
headers={
**self._API_HEADERS,
'X-CSRFToken': csrf_token.value,
'X-Requested-With': 'XMLHttpRequest',
'Referer': url,
}, query={
'query_hash': '9f8827793ef34641b2fb195d4d41151c',
'variables': json.dumps(variables, separators=(',', ':')),
})
media.update(traverse_obj(general_info, ('data', 'shortcode_media')) or {})
if not media:
self.report_warning('General metadata extraction failed', video_id)
self.report_warning('General metadata extraction failed (some metadata might be missing).', video_id)
webpage, urlh = self._download_webpage_handle(url, video_id)
shared_data = self._search_json(
r'window\._sharedData\s*=', webpage, 'shared data', video_id, fatal=False) or {}
info = self._download_json(
f'https://i.instagram.com/api/v1/media/{_id_to_pk(video_id)}/info/', video_id,
fatal=False, note='Downloading video info', errnote=False, headers={
'Accept': '*',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
'Authority': 'www.instagram.com',
'Referer': 'https://www.instagram.com',
'x-ig-app-id': '936619743392459',
})
if info:
media.update(info['items'][0])
return self._extract_product(media)
if shared_data and self._LOGIN_URL not in urlh.geturl():
media.update(traverse_obj(
shared_data, ('entry_data', 'PostPage', 0, 'graphql', 'shortcode_media'),
('entry_data', 'PostPage', 0, 'media'), expected_type=dict) or {})
else:
self.report_warning('Main webpage is locked behind the login page. Retrying with embed webpage')
webpage = self._download_webpage(
f'{url}/embed/', video_id, note='Downloading embed webpage', fatal=False)
additional_data = self._search_json(
r'window\.__additionalDataLoaded\s*\(\s*[^,]+,\s*', webpage, 'additional data', video_id, fatal=False)
if not additional_data:
self.raise_login_required('Requested content is not available, rate-limit reached or login required')
webpage = self._download_webpage(
f'https://www.instagram.com/p/{video_id}/embed/', video_id,
note='Downloading embed webpage', fatal=False)
if not webpage:
self.raise_login_required('Requested content was not found, the content might be private')
product_item = traverse_obj(additional_data, ('items', 0), expected_type=dict)
if product_item:
media.update(product_item)
return self._extract_product(media)
additional_data = self._search_json(
r'window\.__additionalDataLoaded\s*\(\s*[^,]+,\s*', webpage, 'additional data', video_id, fatal=False)
product_item = traverse_obj(additional_data, ('items', 0), expected_type=dict)
if product_item:
media.update(product_item)
return self._extract_product(media)
media.update(traverse_obj(
additional_data, ('graphql', 'shortcode_media'), 'shortcode_media', expected_type=dict) or {})
media.update(traverse_obj(
additional_data, ('graphql', 'shortcode_media'), 'shortcode_media', expected_type=dict) or {})
username = traverse_obj(media, ('owner', 'username')) or self._search_regex(
r'"owner"\s*:\s*{\s*"username"\s*:\s*"(.+?)"', webpage, 'username', fatal=False)
@@ -649,12 +679,8 @@ class InstagramStoryIE(InstagramBaseIE):
story_info_url = user_id if username != 'highlights' else f'highlight:{story_id}'
videos = traverse_obj(self._download_json(
f'https://i.instagram.com/api/v1/feed/reels_media/?reel_ids={story_info_url}',
story_id, errnote=False, fatal=False, headers={
'X-IG-App-ID': 936619743392459,
'X-ASBD-ID': 198387,
'X-IG-WWW-Claim': 0,
}), 'reels')
f'{self._API_BASE_URL}/feed/reels_media/?reel_ids={story_info_url}',
story_id, errnote=False, fatal=False, headers=self._API_HEADERS), 'reels')
if not videos:
self.raise_login_required('You need to log in to access this content')

View File

@@ -1,3 +1,4 @@
import collections
import contextlib
import json
import os
@@ -9,8 +10,10 @@ from ..utils import (
ExtractorError,
Popen,
check_executable,
format_field,
get_exe_version,
is_outdated_version,
shell_quote,
)
@@ -49,7 +52,7 @@ class PhantomJSwrapper:
This class is experimental.
"""
_TEMPLATE = r'''
_BASE_JS = R'''
phantom.onError = function(msg, trace) {{
var msgStack = ['PHANTOM ERROR: ' + msg];
if(trace && trace.length) {{
@@ -62,6 +65,9 @@ class PhantomJSwrapper:
console.error(msgStack.join('\n'));
phantom.exit(1);
}};
'''
_TEMPLATE = R'''
var page = require('webpage').create();
var fs = require('fs');
var read = {{ mode: 'r', charset: 'utf-8' }};
@@ -116,14 +122,18 @@ class PhantomJSwrapper:
'Your copy of PhantomJS is outdated, update it to version '
'%s or newer if you encounter any errors.' % required_version)
self.options = {
'timeout': timeout,
}
for name in self._TMP_FILE_NAMES:
tmp = tempfile.NamedTemporaryFile(delete=False)
tmp.close()
self._TMP_FILES[name] = tmp
self.options = collections.ChainMap({
'timeout': timeout,
}, {
x: self._TMP_FILES[x].name.replace('\\', '\\\\').replace('"', '\\"')
for x in self._TMP_FILE_NAMES
})
def __del__(self):
for name in self._TMP_FILE_NAMES:
with contextlib.suppress(OSError, KeyError):
@@ -194,31 +204,35 @@ class PhantomJSwrapper:
self._save_cookies(url)
replaces = self.options
replaces['url'] = url
user_agent = headers.get('User-Agent') or self.extractor.get_param('http_headers')['User-Agent']
replaces['ua'] = user_agent.replace('"', '\\"')
replaces['jscode'] = jscode
jscode = self._TEMPLATE.format_map(self.options.new_child({
'url': url,
'ua': user_agent.replace('"', '\\"'),
'jscode': jscode,
}))
for x in self._TMP_FILE_NAMES:
replaces[x] = self._TMP_FILES[x].name.replace('\\', '\\\\').replace('"', '\\"')
stdout = self.execute(jscode, video_id, note2)
with open(self._TMP_FILES['script'].name, 'wb') as f:
f.write(self._TEMPLATE.format(**replaces).encode('utf-8'))
if video_id is None:
self.extractor.to_screen(f'{note2}')
else:
self.extractor.to_screen(f'{video_id}: {note2}')
stdout, stderr, returncode = Popen.run(
[self.exe, '--ssl-protocol=any', self._TMP_FILES['script'].name],
text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if returncode:
raise ExtractorError(f'Executing JS failed:\n{stderr}')
with open(self._TMP_FILES['html'].name, 'rb') as f:
html = f.read().decode('utf-8')
self._load_cookies()
return html, stdout
def execute(self, jscode, video_id=None, note='Executing JS'):
"""Execute JS and return stdout"""
if 'phantom.exit();' not in jscode:
jscode += ';\nphantom.exit();'
jscode = self._BASE_JS + jscode
with open(self._TMP_FILES['script'].name, 'w', encoding='utf-8') as f:
f.write(jscode)
self.extractor.to_screen(f'{format_field(video_id, None, "%s: ")}{note}')
cmd = [self.exe, '--ssl-protocol=any', self._TMP_FILES['script'].name]
self.extractor.write_debug(f'PhantomJS command line: {shell_quote(cmd)}')
stdout, stderr, returncode = Popen.run(cmd, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if returncode:
raise ExtractorError(f'Executing JS failed:\n{stderr.strip()}')
return stdout

View File

@@ -156,7 +156,7 @@ class RaiBaseIE(InfoExtractor):
br = int_or_none(tbr)
if len(fmts) == 1 and not br:
br = fmts[0].get('tbr')
if br or 0 > 300:
if br and br > 300:
tbr = compat_str(math.floor(br / 100) * 100)
else:
tbr = '250'

View File

@@ -69,6 +69,10 @@ class RedBeeBaseIE(InfoExtractor):
fmts, subs = self._extract_m3u8_formats_and_subtitles(
format['mediaLocator'], asset_id, fatal=False)
if format.get('drm'):
for f in fmts:
f['has_drm'] = True
formats.extend(fmts)
self._merge_subtitles(subs, target=subtitles)
@@ -269,8 +273,17 @@ class RTBFIE(RedBeeBaseIE):
embed_page = self._download_webpage(
'https://www.rtbf.be/auvio/embed/' + ('direct' if live else 'media'),
media_id, query={'id': media_id})
data = self._parse_json(self._html_search_regex(
r'data-media="([^"]+)"', embed_page, 'media data'), media_id)
media_data = self._html_search_regex(r'data-media="([^"]+)"', embed_page, 'media data', fatal=False)
if not media_data:
if re.search(r'<div[^>]+id="js-error-expired"[^>]+class="(?![^"]*hidden)', embed_page):
raise ExtractorError('Livestream has ended.', expected=True)
if re.search(r'<div[^>]+id="js-sso-connect"[^>]+class="(?![^"]*hidden)', embed_page):
self.raise_login_required()
raise ExtractorError('Could not find media data')
data = self._parse_json(media_data, media_id)
error = data.get('error')
if error:
@@ -280,15 +293,20 @@ class RTBFIE(RedBeeBaseIE):
if provider in self._PROVIDERS:
return self.url_result(data['url'], self._PROVIDERS[provider])
title = data['subtitle']
title = traverse_obj(data, 'subtitle', 'title')
is_live = data.get('isLive')
height_re = r'-(\d+)p\.'
formats = []
formats, subtitles = [], {}
m3u8_url = data.get('urlHlsAes128') or data.get('urlHls')
# The old api still returns m3u8 and mpd manifest for livestreams, but these are 'fake'
# since all they contain is a 20s video that is completely unrelated.
# https://github.com/yt-dlp/yt-dlp/issues/4656#issuecomment-1214461092
m3u8_url = None if data.get('isLive') else traverse_obj(data, 'urlHlsAes128', 'urlHls')
if m3u8_url:
formats.extend(self._extract_m3u8_formats(
m3u8_url, media_id, 'mp4', m3u8_id='hls', fatal=False))
fmts, subs = self._extract_m3u8_formats_and_subtitles(
m3u8_url, media_id, 'mp4', m3u8_id='hls', fatal=False)
formats.extend(fmts)
self._merge_subtitles(subs, target=subtitles)
fix_url = lambda x: x.replace('//rtbf-vod.', '//rtbf.') if '/geo/drm/' in x else x
http_url = data.get('url')
@@ -319,10 +337,12 @@ class RTBFIE(RedBeeBaseIE):
'height': height,
})
mpd_url = data.get('urlDash')
mpd_url = None if data.get('isLive') else data.get('urlDash')
if mpd_url and (self.get_param('allow_unplayable_formats') or not data.get('drm')):
formats.extend(self._extract_mpd_formats(
mpd_url, media_id, mpd_id='dash', fatal=False))
fmts, subs = self._extract_mpd_formats_and_subtitles(
mpd_url, media_id, mpd_id='dash', fatal=False)
formats.extend(fmts)
self._merge_subtitles(subs, target=subtitles)
audio_url = data.get('urlAudio')
if audio_url:
@@ -332,7 +352,6 @@ class RTBFIE(RedBeeBaseIE):
'vcodec': 'none',
})
subtitles = {}
for track in (data.get('tracks') or {}).values():
sub_url = track.get('url')
if not sub_url:
@@ -342,7 +361,7 @@ class RTBFIE(RedBeeBaseIE):
})
if not formats:
fmts, subs = self._get_formats_and_subtitles(url, media_id)
fmts, subs = self._get_formats_and_subtitles(url, f'live_{media_id}' if is_live else media_id)
formats.extend(fmts)
self._merge_subtitles(subs, target=subtitles)

View File

@@ -44,7 +44,7 @@ class SovietsClosetIE(SovietsClosetBaseIE):
_TESTS = [
{
'url': 'https://sovietscloset.com/video/1337',
'md5': '11e58781c4ca5b283307aa54db5b3f93',
'md5': 'bd012b04b261725510ca5383074cdd55',
'info_dict': {
'id': '1337',
'ext': 'mp4',
@@ -69,11 +69,11 @@ class SovietsClosetIE(SovietsClosetBaseIE):
},
{
'url': 'https://sovietscloset.com/video/1105',
'md5': '578b1958a379e7110ba38697042e9efb',
'md5': '89fa928f183893cb65a0b7be846d8a90',
'info_dict': {
'id': '1105',
'ext': 'mp4',
'title': 'Arma 3 - Zeus Games #3',
'title': 'Arma 3 - Zeus Games #5',
'uploader': 'SovietWomble',
'thumbnail': r're:^https?://.*\.b-cdn\.net/c0e5e76f-3a93-40b4-bf01-12343c2eec5d/thumbnail\.jpg$',
'uploader': 'SovietWomble',
@@ -89,8 +89,8 @@ class SovietsClosetIE(SovietsClosetBaseIE):
'availability': 'public',
'series': 'Arma 3',
'season': 'Zeus Games',
'episode_number': 3,
'episode': 'Episode 3',
'episode_number': 5,
'episode': 'Episode 5',
},
},
]
@@ -122,7 +122,7 @@ class SovietsClosetIE(SovietsClosetBaseIE):
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
static_assets_base = self._search_regex(r'staticAssetsBase:\"(.*?)\"', webpage, 'staticAssetsBase')
static_assets_base = self._search_regex(r'(/_nuxt/static/\d+)', webpage, 'staticAssetsBase')
static_assets_base = f'https://sovietscloset.com{static_assets_base}'
stream = self.parse_nuxt_jsonp(f'{static_assets_base}/video/{video_id}/payload.js', video_id, 'video')['stream']
@@ -181,7 +181,7 @@ class SovietsClosetPlaylistIE(SovietsClosetBaseIE):
webpage = self._download_webpage(url, playlist_id)
static_assets_base = self._search_regex(r'staticAssetsBase:\"(.*?)\"', webpage, 'staticAssetsBase')
static_assets_base = self._search_regex(r'(/_nuxt/static/\d+)', webpage, 'staticAssetsBase')
static_assets_base = f'https://sovietscloset.com{static_assets_base}'
sovietscloset = self.parse_nuxt_jsonp(f'{static_assets_base}/payload.js', playlist_id, 'global')['games']

View File

@@ -17,6 +17,7 @@ import urllib.error
import urllib.parse
from .common import InfoExtractor, SearchInfoExtractor
from .openload import PhantomJSwrapper
from ..compat import functools
from ..jsinterp import JSInterpreter
from ..utils import (
@@ -809,7 +810,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
# Youtube sometimes sends incomplete data
# See: https://github.com/ytdl-org/youtube-dl/issues/28194
if not traverse_obj(response, *variadic(check_get_keys)):
retry.error = ExtractorError('Incomplete data received')
retry.error = ExtractorError('Incomplete data received', expected=True)
continue
return response
@@ -867,7 +868,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
else None),
'live_status': ('is_upcoming' if scheduled_timestamp is not None
else 'was_live' if 'streamed' in time_text.lower()
else 'is_live' if overlay_style is not None and overlay_style == 'LIVE' or 'live now' in badges
else 'is_live' if overlay_style == 'LIVE' or 'live now' in badges
else None),
'release_timestamp': scheduled_timestamp,
'availability': self._availability(needs_premium='premium' in badges, needs_subscription='members only' in badges)
@@ -2512,20 +2513,17 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
assert os.path.basename(func_id) == func_id
self.write_debug(f'Extracting signature function {func_id}')
cache_spec = self.cache.load('youtube-sigfuncs', func_id)
if cache_spec is not None:
return lambda s: ''.join(s[i] for i in cache_spec)
cache_spec, code = self.cache.load('youtube-sigfuncs', func_id), None
code = self._load_player(video_id, player_url)
if not cache_spec:
code = self._load_player(video_id, player_url)
if code:
res = self._parse_sig_js(code)
test_string = ''.join(map(chr, range(len(example_sig))))
cache_res = res(test_string)
cache_spec = [ord(c) for c in cache_res]
cache_spec = [ord(c) for c in res(test_string)]
self.cache.store('youtube-sigfuncs', func_id, cache_spec)
return res
return lambda s: ''.join(s[i] for i in cache_spec)
def _print_sig_code(self, func, example_sig):
if not self.get_param('youtube_print_sig_code'):
@@ -2593,18 +2591,29 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
initial_function = jsi.extract_function(funcname)
return lambda s: initial_function([s])
def _cached(self, func, *cache_id):
def inner(*args, **kwargs):
if cache_id not in self._player_cache:
try:
self._player_cache[cache_id] = func(*args, **kwargs)
except ExtractorError as e:
self._player_cache[cache_id] = e
except Exception as e:
self._player_cache[cache_id] = ExtractorError(traceback.format_exc(), cause=e)
ret = self._player_cache[cache_id]
if isinstance(ret, Exception):
raise ret
return ret
return inner
def _decrypt_signature(self, s, video_id, player_url):
"""Turn the encrypted s field into a working signature"""
try:
player_id = (player_url, self._signature_cache_id(s))
if player_id not in self._player_cache:
func = self._extract_signature_function(video_id, player_url, s)
self._player_cache[player_id] = func
func = self._player_cache[player_id]
self._print_sig_code(func, s)
return func(s)
except Exception as e:
raise ExtractorError(traceback.format_exc(), cause=e, video_id=video_id)
extract_sig = self._cached(
self._extract_signature_function, 'sig', player_url, self._signature_cache_id(s))
func = extract_sig(video_id, player_url, s)
self._print_sig_code(func, s)
return func(s)
def _decrypt_nsig(self, s, video_id, player_url):
"""Turn the encrypted n field into a working signature"""
@@ -2612,49 +2621,68 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
raise ExtractorError('Cannot decrypt nsig without player_url')
player_url = urljoin('https://www.youtube.com', player_url)
sig_id = ('nsig_value', s)
if sig_id in self._player_cache:
return self._player_cache[sig_id]
try:
player_id = ('nsig', player_url)
if player_id not in self._player_cache:
self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
func = self._player_cache[player_id]
self._player_cache[sig_id] = func(s)
self.write_debug(f'Decrypted nsig {s} => {self._player_cache[sig_id]}')
return self._player_cache[sig_id]
except Exception as e:
raise ExtractorError(traceback.format_exc(), cause=e, video_id=video_id)
def _extract_n_function_name(self, jscode):
nfunc, idx = self._search_regex(
r'\.get\("n"\)\)&&\(b=(?P<nfunc>[a-zA-Z0-9$]+)(?:\[(?P<idx>\d+)\])?\([a-zA-Z0-9]\)',
jscode, 'Initial JS player n function name', group=('nfunc', 'idx'))
if not idx:
return nfunc
return json.loads(js_to_json(self._search_regex(
rf'var {re.escape(nfunc)}\s*=\s*(\[.+?\]);', jscode,
f'Initial JS player n function list ({nfunc}.{idx})')))[int(idx)]
def _extract_n_function(self, video_id, player_url):
player_id = self._extract_player_info(player_url)
func_code = self.cache.load('youtube-nsig', player_id)
if func_code:
jsi = JSInterpreter(func_code)
else:
jscode = self._load_player(video_id, player_url)
funcname = self._extract_n_function_name(jscode)
jsi = JSInterpreter(jscode)
func_code = jsi.extract_function_code(funcname)
self.cache.store('youtube-nsig', player_id, func_code)
jsi, player_id, func_code = self._extract_n_function_code(video_id, player_url)
if self.get_param('youtube_print_sig_code'):
self.to_screen(f'Extracted nsig function from {player_id}:\n{func_code[1]}\n')
try:
extract_nsig = self._cached(self._extract_n_function_from_code, 'nsig func', player_url)
ret = extract_nsig(jsi, func_code)(s)
except JSInterpreter.Exception as e:
try:
jsi = PhantomJSwrapper(self)
except ExtractorError:
raise e
self.report_warning(
f'Native nsig extraction failed: Trying with PhantomJS\n'
f' n = {s} ; player = {player_url}', video_id)
self.write_debug(e)
args, func_body = func_code
ret = jsi.execute(
f'console.log(function({", ".join(args)}) {{ {func_body} }}({s!r}));',
video_id=video_id, note='Executing signature code').strip()
self.write_debug(f'Decrypted nsig {s} => {ret}')
return ret
def _extract_n_function_code(self, video_id, player_url):
player_id = self._extract_player_info(player_url)
func_code = self.cache.load('youtube-nsig', player_id)
jscode = func_code or self._load_player(video_id, player_url)
jsi = JSInterpreter(jscode)
if func_code:
return jsi, player_id, func_code
funcname, idx = self._search_regex(
r'\.get\("n"\)\)&&\(b=(?P<nfunc>[a-zA-Z0-9$]+)(?:\[(?P<idx>\d+)\])?\([a-zA-Z0-9]\)',
jscode, 'Initial JS player n function name', group=('nfunc', 'idx'))
if idx:
funcname = json.loads(js_to_json(self._search_regex(
rf'var {re.escape(funcname)}\s*=\s*(\[.+?\]);', jscode,
f'Initial JS player n function list ({funcname}.{idx})')))[int(idx)]
func_code = jsi.extract_function_code(funcname)
self.cache.store('youtube-nsig', player_id, func_code)
return jsi, player_id, func_code
def _extract_n_function_from_code(self, jsi, func_code):
func = jsi.extract_function_from_code(*func_code)
return lambda s: func([s])
def extract_nsig(s):
try:
ret = func([s])
except JSInterpreter.Exception:
raise
except Exception as e:
raise JSInterpreter.Exception(traceback.format_exc(), cause=e)
if ret.startswith('enhanced_except_'):
raise JSInterpreter.Exception('Signature function returned an exception')
return ret
return extract_nsig
def _extract_signature_timestamp(self, video_id, player_url, ytcfg=None, fatal=False):
"""
@@ -3168,7 +3196,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
def _extract_formats_and_subtitles(self, streaming_data, video_id, player_url, is_live, duration):
itags, stream_ids = {}, []
itag_qualities, res_qualities = {}, {}
itag_qualities, res_qualities = {}, {0: -1}
q = qualities([
# Normally tiny is the smallest video-only formats. But
# audio-only formats with unknown quality may get tagged as tiny
@@ -3220,7 +3248,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
self._decrypt_signature(encrypted_sig, video_id, player_url)
)
except ExtractorError as e:
self.report_warning('Signature extraction failed: Some formats may be missing', only_once=True)
self.report_warning('Signature extraction failed: Some formats may be missing',
video_id=video_id, only_once=True)
self.write_debug(e, only_once=True)
continue
@@ -3228,12 +3257,17 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
throttled = False
if query.get('n'):
try:
decrypt_nsig = self._cached(self._decrypt_nsig, 'nsig', query['n'][0])
fmt_url = update_url_query(fmt_url, {
'n': self._decrypt_nsig(query['n'][0], video_id, player_url)})
'n': decrypt_nsig(query['n'][0], video_id, player_url)
})
except ExtractorError as e:
phantomjs_hint = ''
if isinstance(e, JSInterpreter.Exception):
phantomjs_hint = f' Install {self._downloader._format_err("PhantomJS", self._downloader.Styles.EMPHASIS)} to workaround the issue\n'
self.report_warning(
'nsig extraction failed: You may experience throttling for some formats\n'
f'n = {query["n"][0]} ; player = {player_url}', only_once=True)
f'nsig extraction failed: You may experience throttling for some formats\n{phantomjs_hint}'
f' n = {query["n"][0]} ; player = {player_url}', video_id=video_id, only_once=True)
self.write_debug(e, only_once=True)
throttled = True
@@ -3320,10 +3354,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
f['format_id'] = itag
itags[itag] = proto
f['quality'] = next((
q(qdict[val])
for val, qdict in ((f.get('format_id', '').split('-')[0], itag_qualities), (f.get('height'), res_qualities))
if val in qdict), -1)
f['quality'] = itag_qualities.get(try_get(f, lambda f: f['format_id'].split('-')[0]), -1)
if f['quality'] == -1 and f.get('height'):
f['quality'] = q(res_qualities[min(res_qualities, key=lambda x: abs(x - f['height']))])
return True
subtitles = {}

View File

@@ -236,32 +236,24 @@ class ZattooPlatformBaseIE(InfoExtractor):
def _real_extract(self, url):
video_id, record_id = self._match_valid_url(url).groups()
return self._extract_video(video_id, record_id)
return getattr(self, f'_extract_{self._TYPE}')(video_id or record_id)
def _make_valid_url(host):
return rf'https?://(?:www\.)?{re.escape(host)}/watch/[^/]+?/(?P<id>[0-9]+)[^/]+(?:/(?P<recid>[0-9]+))?'
def _create_valid_url(host, match, qs, base_re=None):
match_base = fr'|{base_re}/(?P<vid1>{match})' if base_re else '(?P<vid1>)'
return rf'''(?x)https?://(?:www\.)?{re.escape(host)}/(?:
[^?#]+\?(?:[^#]+&)?{qs}=(?P<vid2>{match})
{match_base}
)'''
class ZattooBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'zattoo'
_HOST = 'zattoo.com'
@staticmethod
def _create_valid_url(match, qs, base_re=None):
match_base = fr'|{base_re}/(?P<vid1>{match})' if base_re else '(?P<vid1>)'
return rf'''(?x)https?://(?:www\.)?zattoo\.com/(?:
[^?#]+\?(?:[^#]+&)?{qs}=(?P<vid2>{match})
{match_base}
)'''
def _real_extract(self, url):
vid1, vid2 = self._match_valid_url(url).group('vid1', 'vid2')
return getattr(self, f'_extract_{self._TYPE}')(vid1 or vid2)
class ZattooIE(ZattooBaseIE):
_VALID_URL = ZattooBaseIE._create_valid_url(r'\d+', 'program', '(?:program|watch)/[^/]+')
_VALID_URL = _create_valid_url(ZattooBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://zattoo.com/program/zdf/250170418',
@@ -288,7 +280,7 @@ class ZattooIE(ZattooBaseIE):
class ZattooLiveIE(ZattooBaseIE):
_VALID_URL = ZattooBaseIE._create_valid_url(r'[^/?&#]+', 'channel', 'live')
_VALID_URL = _create_valid_url(ZattooBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://zattoo.com/channels/german?channel=srf_zwei',
@@ -304,7 +296,7 @@ class ZattooLiveIE(ZattooBaseIE):
class ZattooMoviesIE(ZattooBaseIE):
_VALID_URL = ZattooBaseIE._create_valid_url(r'\w+', 'movie_id', 'vod/movies')
_VALID_URL = _create_valid_url(ZattooBaseIE._HOST, r'\w+', 'movie_id', 'vod/movies')
_TYPE = 'ondemand'
_TESTS = [{
'url': 'https://zattoo.com/vod/movies/7521',
@@ -316,7 +308,7 @@ class ZattooMoviesIE(ZattooBaseIE):
class ZattooRecordingsIE(ZattooBaseIE):
_VALID_URL = ZattooBaseIE._create_valid_url(r'\d+', 'recording')
_VALID_URL = _create_valid_url('zattoo.com', r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://zattoo.com/recordings?recording=193615508',
@@ -327,139 +319,547 @@ class ZattooRecordingsIE(ZattooBaseIE):
}]
class NetPlusIE(ZattooPlatformBaseIE):
class NetPlusTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'netplus'
_HOST = 'netplus.tv'
_API_HOST = 'www.%s' % _HOST
_VALID_URL = _make_valid_url(_HOST)
class NetPlusTVIE(NetPlusTVBaseIE):
_VALID_URL = _create_valid_url(NetPlusTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://www.netplus.tv/watch/abc/123-abc',
'url': 'https://netplus.tv/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://netplus.tv/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class MNetTVIE(ZattooPlatformBaseIE):
class NetPlusTVLiveIE(NetPlusTVBaseIE):
_VALID_URL = _create_valid_url(NetPlusTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://netplus.tv/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://netplus.tv/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if NetPlusTVIE.suitable(url) else super().suitable(url)
class NetPlusTVRecordingsIE(NetPlusTVBaseIE):
_VALID_URL = _create_valid_url(NetPlusTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://netplus.tv/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://netplus.tv/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class MNetTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'mnettv'
_HOST = 'tvplus.m-net.de'
_VALID_URL = _make_valid_url(_HOST)
class MNetTVIE(MNetTVBaseIE):
_VALID_URL = _create_valid_url(MNetTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://tvplus.m-net.de/watch/abc/123-abc',
'url': 'https://tvplus.m-net.de/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://tvplus.m-net.de/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class WalyTVIE(ZattooPlatformBaseIE):
class MNetTVLiveIE(MNetTVBaseIE):
_VALID_URL = _create_valid_url(MNetTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://tvplus.m-net.de/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://tvplus.m-net.de/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if MNetTVIE.suitable(url) else super().suitable(url)
class MNetTVRecordingsIE(MNetTVBaseIE):
_VALID_URL = _create_valid_url(MNetTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://tvplus.m-net.de/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://tvplus.m-net.de/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class WalyTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'walytv'
_HOST = 'player.waly.tv'
_VALID_URL = _make_valid_url(_HOST)
class WalyTVIE(WalyTVBaseIE):
_VALID_URL = _create_valid_url(WalyTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://player.waly.tv/watch/abc/123-abc',
'url': 'https://player.waly.tv/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://player.waly.tv/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class BBVTVIE(ZattooPlatformBaseIE):
class WalyTVLiveIE(WalyTVBaseIE):
_VALID_URL = _create_valid_url(WalyTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://player.waly.tv/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://player.waly.tv/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if WalyTVIE.suitable(url) else super().suitable(url)
class WalyTVRecordingsIE(WalyTVBaseIE):
_VALID_URL = _create_valid_url(WalyTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://player.waly.tv/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://player.waly.tv/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class BBVTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'bbvtv'
_HOST = 'bbv-tv.net'
_API_HOST = 'www.%s' % _HOST
_VALID_URL = _make_valid_url(_HOST)
class BBVTVIE(BBVTVBaseIE):
_VALID_URL = _create_valid_url(BBVTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://www.bbv-tv.net/watch/abc/123-abc',
'url': 'https://bbv-tv.net/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://bbv-tv.net/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class VTXTVIE(ZattooPlatformBaseIE):
class BBVTVLiveIE(BBVTVBaseIE):
_VALID_URL = _create_valid_url(BBVTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://bbv-tv.net/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://bbv-tv.net/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if BBVTVIE.suitable(url) else super().suitable(url)
class BBVTVRecordingsIE(BBVTVBaseIE):
_VALID_URL = _create_valid_url(BBVTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://bbv-tv.net/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://bbv-tv.net/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class VTXTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'vtxtv'
_HOST = 'vtxtv.ch'
_API_HOST = 'www.%s' % _HOST
_VALID_URL = _make_valid_url(_HOST)
class VTXTVIE(VTXTVBaseIE):
_VALID_URL = _create_valid_url(VTXTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://www.vtxtv.ch/watch/abc/123-abc',
'url': 'https://vtxtv.ch/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://vtxtv.ch/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class GlattvisionTVIE(ZattooPlatformBaseIE):
class VTXTVLiveIE(VTXTVBaseIE):
_VALID_URL = _create_valid_url(VTXTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://vtxtv.ch/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://vtxtv.ch/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if VTXTVIE.suitable(url) else super().suitable(url)
class VTXTVRecordingsIE(VTXTVBaseIE):
_VALID_URL = _create_valid_url(VTXTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://vtxtv.ch/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://vtxtv.ch/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class GlattvisionTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'glattvisiontv'
_HOST = 'iptv.glattvision.ch'
_VALID_URL = _make_valid_url(_HOST)
class GlattvisionTVIE(GlattvisionTVBaseIE):
_VALID_URL = _create_valid_url(GlattvisionTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://iptv.glattvision.ch/watch/abc/123-abc',
'url': 'https://iptv.glattvision.ch/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://iptv.glattvision.ch/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class SAKTVIE(ZattooPlatformBaseIE):
class GlattvisionTVLiveIE(GlattvisionTVBaseIE):
_VALID_URL = _create_valid_url(GlattvisionTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://iptv.glattvision.ch/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://iptv.glattvision.ch/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if GlattvisionTVIE.suitable(url) else super().suitable(url)
class GlattvisionTVRecordingsIE(GlattvisionTVBaseIE):
_VALID_URL = _create_valid_url(GlattvisionTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://iptv.glattvision.ch/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://iptv.glattvision.ch/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class SAKTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'saktv'
_HOST = 'saktv.ch'
_API_HOST = 'www.%s' % _HOST
_VALID_URL = _make_valid_url(_HOST)
class SAKTVIE(SAKTVBaseIE):
_VALID_URL = _create_valid_url(SAKTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://www.saktv.ch/watch/abc/123-abc',
'url': 'https://saktv.ch/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://saktv.ch/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class EWETVIE(ZattooPlatformBaseIE):
class SAKTVLiveIE(SAKTVBaseIE):
_VALID_URL = _create_valid_url(SAKTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://saktv.ch/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://saktv.ch/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if SAKTVIE.suitable(url) else super().suitable(url)
class SAKTVRecordingsIE(SAKTVBaseIE):
_VALID_URL = _create_valid_url(SAKTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://saktv.ch/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://saktv.ch/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class EWETVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'ewetv'
_HOST = 'tvonline.ewe.de'
_VALID_URL = _make_valid_url(_HOST)
class EWETVIE(EWETVBaseIE):
_VALID_URL = _create_valid_url(EWETVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://tvonline.ewe.de/watch/abc/123-abc',
'url': 'https://tvonline.ewe.de/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://tvonline.ewe.de/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class QuantumTVIE(ZattooPlatformBaseIE):
class EWETVLiveIE(EWETVBaseIE):
_VALID_URL = _create_valid_url(EWETVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://tvonline.ewe.de/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://tvonline.ewe.de/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if EWETVIE.suitable(url) else super().suitable(url)
class EWETVRecordingsIE(EWETVBaseIE):
_VALID_URL = _create_valid_url(EWETVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://tvonline.ewe.de/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://tvonline.ewe.de/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class QuantumTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'quantumtv'
_HOST = 'quantum-tv.com'
_API_HOST = 'www.%s' % _HOST
_VALID_URL = _make_valid_url(_HOST)
class QuantumTVIE(QuantumTVBaseIE):
_VALID_URL = _create_valid_url(QuantumTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://www.quantum-tv.com/watch/abc/123-abc',
'url': 'https://quantum-tv.com/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://quantum-tv.com/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class OsnatelTVIE(ZattooPlatformBaseIE):
class QuantumTVLiveIE(QuantumTVBaseIE):
_VALID_URL = _create_valid_url(QuantumTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://quantum-tv.com/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://quantum-tv.com/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if QuantumTVIE.suitable(url) else super().suitable(url)
class QuantumTVRecordingsIE(QuantumTVBaseIE):
_VALID_URL = _create_valid_url(QuantumTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://quantum-tv.com/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://quantum-tv.com/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class OsnatelTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'osnateltv'
_HOST = 'tvonline.osnatel.de'
_VALID_URL = _make_valid_url(_HOST)
class OsnatelTVIE(OsnatelTVBaseIE):
_VALID_URL = _create_valid_url(OsnatelTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://tvonline.osnatel.de/watch/abc/123-abc',
'url': 'https://tvonline.osnatel.de/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://tvonline.osnatel.de/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class EinsUndEinsTVIE(ZattooPlatformBaseIE):
class OsnatelTVLiveIE(OsnatelTVBaseIE):
_VALID_URL = _create_valid_url(OsnatelTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://tvonline.osnatel.de/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://tvonline.osnatel.de/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if OsnatelTVIE.suitable(url) else super().suitable(url)
class OsnatelTVRecordingsIE(OsnatelTVBaseIE):
_VALID_URL = _create_valid_url(OsnatelTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://tvonline.osnatel.de/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://tvonline.osnatel.de/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class EinsUndEinsTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = '1und1tv'
_HOST = '1und1.tv'
_API_HOST = 'www.%s' % _HOST
_VALID_URL = _make_valid_url(_HOST)
class EinsUndEinsTVIE(EinsUndEinsTVBaseIE):
_VALID_URL = _create_valid_url(EinsUndEinsTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://www.1und1.tv/watch/abc/123-abc',
'url': 'https://1und1.tv/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://1und1.tv/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class SaltTVIE(ZattooPlatformBaseIE):
class EinsUndEinsTVLiveIE(EinsUndEinsTVBaseIE):
_VALID_URL = _create_valid_url(EinsUndEinsTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://1und1.tv/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://1und1.tv/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if EinsUndEinsTVIE.suitable(url) else super().suitable(url)
class EinsUndEinsTVRecordingsIE(EinsUndEinsTVBaseIE):
_VALID_URL = _create_valid_url(EinsUndEinsTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://1und1.tv/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://1und1.tv/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]
class SaltTVBaseIE(ZattooPlatformBaseIE):
_NETRC_MACHINE = 'salttv'
_HOST = 'tv.salt.ch'
_VALID_URL = _make_valid_url(_HOST)
class SaltTVIE(SaltTVBaseIE):
_VALID_URL = _create_valid_url(SaltTVBaseIE._HOST, r'\d+', 'program', '(?:program|watch)/[^/]+')
_TYPE = 'video'
_TESTS = [{
'url': 'https://tv.salt.ch/watch/abc/123-abc',
'url': 'https://tv.salt.ch/program/daserste/210177916',
'only_matching': True,
}, {
'url': 'https://tv.salt.ch/guide/german?channel=srf1&program=169860555',
'only_matching': True,
}]
class SaltTVLiveIE(SaltTVBaseIE):
_VALID_URL = _create_valid_url(SaltTVBaseIE._HOST, r'[^/?&#]+', 'channel', 'live')
_TYPE = 'live'
_TESTS = [{
'url': 'https://tv.salt.ch/channels/german?channel=srf_zwei',
'only_matching': True,
}, {
'url': 'https://tv.salt.ch/live/srf1',
'only_matching': True,
}]
@classmethod
def suitable(cls, url):
return False if SaltTVIE.suitable(url) else super().suitable(url)
class SaltTVRecordingsIE(SaltTVBaseIE):
_VALID_URL = _create_valid_url(SaltTVBaseIE._HOST, r'\d+', 'recording')
_TYPE = 'record'
_TESTS = [{
'url': 'https://tv.salt.ch/recordings?recording=193615508',
'only_matching': True,
}, {
'url': 'https://tv.salt.ch/tc/ptc_recordings_all_recordings?recording=193615420',
'only_matching': True,
}]

View File

@@ -16,50 +16,69 @@ from .utils import (
write_string,
)
_NAME_RE = r'[a-zA-Z_$][\w$]*'
# Ref: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence
_OPERATORS = { # None => Defined in JSInterpreter._operator
'?': None,
def _js_bit_op(op):
def wrapped(a, b):
def zeroise(x):
return 0 if x in (None, JS_Undefined) else x
return op(zeroise(a), zeroise(b))
'||': None,
'&&': None,
'&': operator.and_,
'|': operator.or_,
'^': operator.xor,
'===': operator.is_,
'!==': operator.is_not,
'==': operator.eq,
'!=': operator.ne,
'<=': operator.le,
'>=': operator.ge,
'<': operator.lt,
'>': operator.gt,
'>>': operator.rshift,
'<<': operator.lshift,
'+': operator.add,
'-': operator.sub,
'*': operator.mul,
'/': operator.truediv,
'%': operator.mod,
'**': operator.pow,
}
_COMP_OPERATORS = {'===', '!==', '==', '!=', '<=', '>=', '<', '>'}
_MATCHING_PARENS = dict(zip('({[', ')}]'))
_QUOTES = '\'"'
return wrapped
def _ternary(cndn, if_true=True, if_false=False):
def _js_arith_op(op):
def wrapped(a, b):
if JS_Undefined in (a, b):
return float('nan')
return op(a or 0, b or 0)
return wrapped
def _js_div(a, b):
if JS_Undefined in (a, b) or not (a and b):
return float('nan')
return (a or 0) / b if b else float('inf')
def _js_mod(a, b):
if JS_Undefined in (a, b) or not b:
return float('nan')
return (a or 0) % b
def _js_exp(a, b):
if not b:
return 1 # even 0 ** 0 !!
elif JS_Undefined in (a, b):
return float('nan')
return (a or 0) ** b
def _js_eq_op(op):
def wrapped(a, b):
if {a, b} <= {None, JS_Undefined}:
return op(a, a)
return op(a, b)
return wrapped
def _js_comp_op(op):
def wrapped(a, b):
if JS_Undefined in (a, b):
return False
return op(a or 0, b or 0)
return wrapped
def _js_ternary(cndn, if_true=True, if_false=False):
"""Simulate JS's ternary operator (cndn?if_true:if_false)"""
if cndn in (False, None, 0, ''):
if cndn in (False, None, 0, '', JS_Undefined):
return if_false
with contextlib.suppress(TypeError):
if math.isnan(cndn): # NB: NaN cannot be checked by membership
@@ -67,6 +86,50 @@ def _ternary(cndn, if_true=True, if_false=False):
return if_true
# Ref: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence
_OPERATORS = { # None => Defined in JSInterpreter._operator
'?': None,
'??': None,
'||': None,
'&&': None,
'|': _js_bit_op(operator.or_),
'^': _js_bit_op(operator.xor),
'&': _js_bit_op(operator.and_),
'===': operator.is_,
'==': _js_eq_op(operator.eq),
'!==': operator.is_not,
'!=': _js_eq_op(operator.ne),
'<=': _js_comp_op(operator.le),
'>=': _js_comp_op(operator.ge),
'<': _js_comp_op(operator.lt),
'>': _js_comp_op(operator.gt),
'>>': _js_bit_op(operator.rshift),
'<<': _js_bit_op(operator.lshift),
'+': _js_arith_op(operator.add),
'-': _js_arith_op(operator.sub),
'*': _js_arith_op(operator.mul),
'/': _js_div,
'%': _js_mod,
'**': _js_exp,
}
_COMP_OPERATORS = {'===', '!==', '==', '!=', '<=', '>=', '<', '>'}
_NAME_RE = r'[a-zA-Z_$][\w$]*'
_MATCHING_PARENS = dict(zip(*zip('()', '{}', '[]')))
_QUOTES = '\'"/'
class JS_Undefined:
pass
class JS_Break(ExtractorError):
def __init__(self):
ExtractorError.__init__(self, 'Invalid break')
@@ -77,6 +140,12 @@ class JS_Continue(ExtractorError):
ExtractorError.__init__(self, 'Invalid continue')
class JS_Throw(ExtractorError):
def __init__(self, e):
self.error = e
ExtractorError.__init__(self, f'Uncaught exception {e}')
class LocalNameSpace(collections.ChainMap):
def __setitem__(self, key, value):
for scope in self.maps:
@@ -113,6 +182,21 @@ class Debugger:
class JSInterpreter:
__named_object_counter = 0
_RE_FLAGS = {
# special knowledge: Python's re flags are bitmask values, current max 128
# invent new bitmask values well above that for literal parsing
# TODO: new pattern class to execute matches with these flags
'd': 1024, # Generate indices for substring matches
'g': 2048, # Global search
'i': re.I, # Case-insensitive search
'm': re.M, # Multi-line search
's': re.S, # Allows . to match newline characters
'u': re.U, # Treat a pattern as a sequence of unicode code points
'y': 4096, # Perform a "sticky" search that matches starting at the current position in the target string
}
_EXC_NAME = '__yt_dlp_exception__'
def __init__(self, code, objects=None):
self.code, self._functions = code, {}
self._objects = {} if objects is None else objects
@@ -129,21 +213,37 @@ class JSInterpreter:
namespace[name] = obj
return name
@classmethod
def _regex_flags(cls, expr):
flags = 0
if not expr:
return flags, expr
for idx, ch in enumerate(expr):
if ch not in cls._RE_FLAGS:
break
flags |= cls._RE_FLAGS[ch]
return flags, expr[idx + 1:]
@staticmethod
def _separate(expr, delim=',', max_split=None):
OP_CHARS = '+-*/%&|^=<>!,;'
if not expr:
return
counters = {k: 0 for k in _MATCHING_PARENS.values()}
start, splits, pos, delim_len = 0, 0, 0, len(delim) - 1
in_quote, escaping = None, False
in_quote, escaping, after_op, in_regex_char_group = None, False, True, False
for idx, char in enumerate(expr):
if not in_quote and char in _MATCHING_PARENS:
counters[_MATCHING_PARENS[char]] += 1
elif not in_quote and char in counters:
counters[char] -= 1
elif not escaping and char in _QUOTES and in_quote in (char, None):
in_quote = None if in_quote else char
if in_quote or after_op or char != '/':
in_quote = None if in_quote and not in_regex_char_group else char
elif in_quote == '/' and char in '[]':
in_regex_char_group = char == '['
escaping = not escaping and in_quote and char == '\\'
after_op = not in_quote and char in OP_CHARS or (char == ' ' and after_op)
if char != delim[pos] or any(counters.values()) or in_quote:
pos = 0
@@ -167,10 +267,13 @@ class JSInterpreter:
def _operator(self, op, left_val, right_expr, expr, local_vars, allow_recursion):
if op in ('||', '&&'):
if (op == '&&') ^ _ternary(left_val):
if (op == '&&') ^ _js_ternary(left_val):
return left_val # short circuiting
elif op == '??':
if left_val not in (None, JS_Undefined):
return left_val
elif op == '?':
right_expr = _ternary(left_val, *self._separate(right_expr, ':', 1))
right_expr = _js_ternary(left_val, *self._separate(right_expr, ':', 1))
right_val = self.interpret_expression(right_expr, local_vars, allow_recursion)
if not _OPERATORS.get(op):
@@ -181,12 +284,14 @@ class JSInterpreter:
except Exception as e:
raise self.Exception(f'Failed to evaluate {left_val!r} {op} {right_val!r}', expr, cause=e)
def _index(self, obj, idx):
def _index(self, obj, idx, allow_undefined=False):
if idx == 'length':
return len(obj)
try:
return obj[int(idx)] if isinstance(obj, list) else obj[idx]
except Exception as e:
if allow_undefined:
return JS_Undefined
raise self.Exception(f'Cannot get index {idx}', repr(obj), cause=e)
def _dump(self, obj, namespace):
@@ -210,16 +315,22 @@ class JSInterpreter:
if should_return:
return ret, should_return
m = re.match(r'(?P<var>(?:var|const|let)\s)|return(?:\s+|$)', stmt)
m = re.match(r'(?P<var>(?:var|const|let)\s)|return(?:\s+|(?=["\'])|$)|(?P<throw>throw\s+)', stmt)
if m:
expr = stmt[len(m.group(0)):].strip()
if m.group('throw'):
raise JS_Throw(self.interpret_expression(expr, local_vars, allow_recursion))
should_return = not m.group('var')
if not expr:
return None, should_return
if expr[0] in _QUOTES:
inner, outer = self._separate(expr, expr[0], 1)
inner = json.loads(js_to_json(f'{inner}{expr[0]}', strict=True))
if expr[0] == '/':
flags, outer = self._regex_flags(outer)
inner = re.compile(inner[1:], flags=flags)
else:
inner = json.loads(js_to_json(f'{inner}{expr[0]}', strict=True))
if not outer:
return inner, should_return
expr = self._named_object(local_vars, inner) + outer
@@ -242,6 +353,17 @@ class JSInterpreter:
if expr.startswith('{'):
inner, outer = self._separate_at_paren(expr, '}')
# Look for Map first
sub_expressions = [list(self._separate(sub_expr.strip(), ':', 1)) for sub_expr in self._separate(inner)]
if all(len(sub_expr) == 2 for sub_expr in sub_expressions):
def dict_item(key, val):
val = self.interpret_expression(val, local_vars, allow_recursion)
if re.match(_NAME_RE, key):
return key, val
return self.interpret_expression(key, local_vars, allow_recursion), val
return dict(dict_item(k, v) for k, v in sub_expressions), should_return
inner, should_abort = self.interpret_statement(inner, local_vars, allow_recursion)
if not outer or should_abort:
return inner, should_abort or should_return
@@ -263,21 +385,36 @@ class JSInterpreter:
for item in self._separate(inner)])
expr = name + outer
m = re.match(r'(?P<try>try|finally)\s*|(?:(?P<catch>catch)|(?P<for>for)|(?P<switch>switch))\s*\(', expr)
m = re.match(rf'''(?x)
(?P<try>try|finally)\s*|
(?P<catch>catch\s*(?P<err>\(\s*{_NAME_RE}\s*\)))|
(?P<switch>switch)\s*\(|
(?P<for>for)\s*\(|''', expr)
if m and m.group('try'):
if expr[m.end()] == '{':
try_expr, expr = self._separate_at_paren(expr[m.end():], '}')
else:
try_expr, expr = expr[m.end() - 1:], ''
ret, should_abort = self.interpret_statement(try_expr, local_vars, allow_recursion)
if should_abort:
return ret, True
try:
ret, should_abort = self.interpret_statement(try_expr, local_vars, allow_recursion)
if should_abort:
return ret, True
except JS_Throw as e:
local_vars[self._EXC_NAME] = e.error
except Exception as e:
# XXX: This works for now, but makes debugging future issues very hard
local_vars[self._EXC_NAME] = e
ret, should_abort = self.interpret_statement(expr, local_vars, allow_recursion)
return ret, should_abort or should_return
elif m and m.group('catch'):
# We ignore the catch block
_, expr = self._separate_at_paren(expr, '}')
catch_expr, expr = self._separate_at_paren(expr[m.end():], '}')
if self._EXC_NAME in local_vars:
catch_vars = local_vars.new_child({m.group('err'): local_vars.pop(self._EXC_NAME)})
ret, should_abort = self.interpret_statement(catch_expr, catch_vars, allow_recursion)
if should_abort:
return ret, True
ret, should_abort = self.interpret_statement(expr, local_vars, allow_recursion)
return ret, should_abort or should_return
@@ -296,7 +433,7 @@ class JSInterpreter:
start, cndn, increment = self._separate(constructor, ';')
self.interpret_expression(start, local_vars, allow_recursion)
while True:
if not _ternary(self.interpret_expression(cndn, local_vars, allow_recursion)):
if not _js_ternary(self.interpret_expression(cndn, local_vars, allow_recursion)):
break
try:
ret, should_abort = self.interpret_statement(body, local_vars, allow_recursion)
@@ -339,11 +476,12 @@ class JSInterpreter:
# Comma separated statements
sub_expressions = list(self._separate(expr))
expr = sub_expressions.pop().strip() if sub_expressions else ''
for sub_expr in sub_expressions:
ret, should_abort = self.interpret_statement(sub_expr, local_vars, allow_recursion)
if should_abort:
return ret, True
if len(sub_expressions) > 1:
for sub_expr in sub_expressions:
ret, should_abort = self.interpret_statement(sub_expr, local_vars, allow_recursion)
if should_abort:
return ret, True
return ret, False
for m in re.finditer(rf'''(?x)
(?P<pre_sign>\+\+|--)(?P<var1>{_NAME_RE})|
@@ -364,13 +502,13 @@ class JSInterpreter:
(?P<assign>
(?P<out>{_NAME_RE})(?:\[(?P<index>[^\]]+?)\])?\s*
(?P<op>{"|".join(map(re.escape, set(_OPERATORS) - _COMP_OPERATORS))})?
=(?P<expr>.*)$
=(?!=)(?P<expr>.*)$
)|(?P<return>
(?!if|return|true|false|null|undefined)(?P<name>{_NAME_RE})$
)|(?P<indexing>
(?P<in>{_NAME_RE})\[(?P<idx>.+)\]$
)|(?P<attribute>
(?P<var>{_NAME_RE})(?:\.(?P<member>[^(]+)|\[(?P<member2>[^\]]+)\])\s*
(?P<var>{_NAME_RE})(?:(?P<nullish>\?)?\.(?P<member>[^(]+)|\[(?P<member2>[^\]]+)\])\s*
)|(?P<function>
(?P<fname>{_NAME_RE})\((?P<args>.*)\)$
)''', expr)
@@ -381,7 +519,7 @@ class JSInterpreter:
local_vars[m.group('out')] = self._operator(
m.group('op'), left_val, m.group('expr'), expr, local_vars, allow_recursion)
return local_vars[m.group('out')], should_return
elif left_val is None:
elif left_val in (None, JS_Undefined):
raise self.Exception(f'Cannot index undefined variable {m.group("out")}', expr)
idx = self.interpret_expression(m.group('index'), local_vars, allow_recursion)
@@ -389,7 +527,7 @@ class JSInterpreter:
raise self.Exception(f'List index {idx} must be integer', expr)
idx = int(idx)
left_val[idx] = self._operator(
m.group('op'), left_val[idx], m.group('expr'), expr, local_vars, allow_recursion)
m.group('op'), self._index(left_val, idx), m.group('expr'), expr, local_vars, allow_recursion)
return left_val[idx], should_return
elif expr.isdigit():
@@ -399,9 +537,11 @@ class JSInterpreter:
raise JS_Break()
elif expr == 'continue':
raise JS_Continue()
elif expr == 'undefined':
return JS_Undefined, should_return
elif m and m.group('return'):
return local_vars[m.group('name')], should_return
return local_vars.get(m.group('name'), JS_Undefined), should_return
with contextlib.suppress(ValueError):
return json.loads(js_to_json(expr, strict=True)), should_return
@@ -414,20 +554,21 @@ class JSInterpreter:
for op in _OPERATORS:
separated = list(self._separate(expr, op))
right_expr = separated.pop()
while op in '<>*-' and len(separated) > 1 and not separated[-1].strip():
separated.pop()
while True:
if op in '?<>*-' and len(separated) > 1 and not separated[-1].strip():
separated.pop()
elif not (separated and op == '?' and right_expr.startswith('.')):
break
right_expr = f'{op}{right_expr}'
if op != '-':
right_expr = f'{separated.pop()}{op}{right_expr}'
if not separated:
continue
left_val = self.interpret_expression(op.join(separated), local_vars, allow_recursion)
return self._operator(op, 0 if left_val is None else left_val,
right_expr, expr, local_vars, allow_recursion), should_return
return self._operator(op, left_val, right_expr, expr, local_vars, allow_recursion), should_return
if m and m.group('attribute'):
variable = m.group('var')
member = m.group('member')
variable, member, nullish = m.group('var', 'member', 'nullish')
if not member:
member = self.interpret_expression(m.group('member2'), local_vars, allow_recursion)
arg_str = expr[m.end():]
@@ -454,12 +595,19 @@ class JSInterpreter:
obj = local_vars.get(variable, types.get(variable, NO_DEFAULT))
if obj is NO_DEFAULT:
if variable not in self._objects:
self._objects[variable] = self.extract_object(variable)
obj = self._objects[variable]
try:
self._objects[variable] = self.extract_object(variable)
except self.Exception:
if not nullish:
raise
obj = self._objects.get(variable, JS_Undefined)
if nullish and obj is JS_Undefined:
return JS_Undefined
# Member access
if arg_str is None:
return self._index(obj, member)
return self._index(obj, member, nullish)
# Function call
argvals = [

View File

@@ -1,5 +1,4 @@
import collections
import contextvars
import itertools
import json
import os
@@ -83,8 +82,6 @@ class FFmpegPostProcessorError(PostProcessingError):
class FFmpegPostProcessor(PostProcessor):
_ffmpeg_location = contextvars.ContextVar('ffmpeg_location', default=None)
def __init__(self, downloader=None):
PostProcessor.__init__(self, downloader)
self._prefer_ffmpeg = self.get_param('prefer_ffmpeg', True)
@@ -104,7 +101,7 @@ class FFmpegPostProcessor(PostProcessor):
def _determine_executables(self):
programs = [*self._ffmpeg_to_avconv.keys(), *self._ffmpeg_to_avconv.values()]
location = self.get_param('ffmpeg_location', self._ffmpeg_location.get())
location = self.get_param('ffmpeg_location')
if location is None:
return {p: p for p in programs}

View File

@@ -1,4 +1,3 @@
import asyncio
import atexit
import base64
import binascii
@@ -47,7 +46,7 @@ import urllib.request
import xml.etree.ElementTree
import zlib
from .compat import functools # isort: split
from .compat import asyncio, functools # isort: split
from .compat import (
compat_etree_fromstring,
compat_expanduser,
@@ -5764,7 +5763,7 @@ class RetryManager:
if not count:
return warn(e)
elif isinstance(e, ExtractorError):
e = remove_end(str(e.cause) or e.orig_msg, '.')
e = remove_end(str_or_none(e.cause) or e.orig_msg, '.')
warn(f'{e}. Retrying{format_field(suffix, None, " %s")} ({count}/{retries})...')
delay = float_or_none(sleep_func(n=count - 1)) if callable(sleep_func) else sleep_func

View File

@@ -1,8 +1,8 @@
# Autogenerated by devscripts/update-version.py
__version__ = '2022.08.14'
__version__ = '2022.08.18.36'
RELEASE_GIT_HEAD = '55937202b'
RELEASE_GIT_HEAD = '48c88e088'
VARIANT = None

View File

@@ -9,8 +9,8 @@ in RFC 8216 §3.5 <https://tools.ietf.org/html/rfc8216#section-3.5>.
"""
import io
import re
from .compat import re
from .utils import int_or_none, timetuple_from_msec