mirror of
https://github.com/yt-dlp/yt-dlp.git
synced 2024-08-24 16:21:40 +02:00
Improve plugin architecture (#5553)
to make plugins easier to develop and use: * Plugins are now loaded as namespace packages. * Plugins can be loaded in any distribution of yt-dlp (binary, pip, source, etc.). * Plugin packages can be installed and managed via pip, or dropped into any of the documented locations. * Users do not need to edit any code files to install plugins. * Backwards-compatible with previous plugin architecture. As a side-effect, yt-dlp will now search in a few more locations for config files. Closes https://github.com/yt-dlp/yt-dlp/issues/1389 Authored by: flashdagger, coletdjnz, pukkandan, Grub4K Co-authored-by: Marcel <flashdagger@googlemail.com> Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com> Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
This commit is contained in:
parent
2fb0f85868
commit
8e40b9d1ec
8
.gitignore
vendored
8
.gitignore
vendored
@ -120,9 +120,5 @@ yt-dlp.zip
|
||||
*/extractor/lazy_extractors.py
|
||||
|
||||
# Plugins
|
||||
ytdlp_plugins/extractor/*
|
||||
!ytdlp_plugins/extractor/__init__.py
|
||||
!ytdlp_plugins/extractor/sample.py
|
||||
ytdlp_plugins/postprocessor/*
|
||||
!ytdlp_plugins/postprocessor/__init__.py
|
||||
!ytdlp_plugins/postprocessor/sample.py
|
||||
ytdlp_plugins/*
|
||||
yt-dlp-plugins/*
|
||||
|
66
README.md
66
README.md
@ -61,6 +61,8 @@
|
||||
* [Modifying metadata examples](#modifying-metadata-examples)
|
||||
* [EXTRACTOR ARGUMENTS](#extractor-arguments)
|
||||
* [PLUGINS](#plugins)
|
||||
* [Installing Plugins](#installing-plugins)
|
||||
* [Developing Plugins](#developing-plugins)
|
||||
* [EMBEDDING YT-DLP](#embedding-yt-dlp)
|
||||
* [Embedding examples](#embedding-examples)
|
||||
* [DEPRECATED OPTIONS](#deprecated-options)
|
||||
@ -1110,15 +1112,20 @@ # CONFIGURATION
|
||||
* If `-P` is not given, the current directory is searched
|
||||
1. **User Configuration**:
|
||||
* `${XDG_CONFIG_HOME}/yt-dlp/config` (recommended on Linux/macOS)
|
||||
* `${XDG_CONFIG_HOME}/yt-dlp/config.txt`
|
||||
* `${XDG_CONFIG_HOME}/yt-dlp.conf`
|
||||
* `${APPDATA}/yt-dlp/config` (recommended on Windows)
|
||||
* `${APPDATA}/yt-dlp/config.txt`
|
||||
* `~/yt-dlp.conf`
|
||||
* `~/yt-dlp.conf.txt`
|
||||
* `~/.yt-dlp/config`
|
||||
* `~/.yt-dlp/config.txt`
|
||||
|
||||
See also: [Notes about environment variables](#notes-about-environment-variables)
|
||||
1. **System Configuration**:
|
||||
* `/etc/yt-dlp.conf`
|
||||
* `/etc/yt-dlp/config`
|
||||
* `/etc/yt-dlp/config.txt`
|
||||
|
||||
E.g. with the following configuration file yt-dlp will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory:
|
||||
```
|
||||
@ -1789,19 +1796,68 @@ #### twitter
|
||||
|
||||
# PLUGINS
|
||||
|
||||
Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`; where `<root-dir>` is the directory of the binary (`<root-dir>/yt-dlp`), or the root directory of the module if you are running directly from source-code (`<root dir>/yt_dlp/__main__.py`). Plugins are currently not supported for the `pip` version
|
||||
Note that **all** plugins are imported even if not invoked, and that **there are no checks** performed on plugin code. **Use plugins at your own risk and only if you trust the code!**
|
||||
|
||||
Plugins can be of `<type>`s `extractor` or `postprocessor`. Extractor plugins do not need to be enabled from the CLI and are automatically invoked when the input URL is suitable for it. Postprocessor plugins can be invoked using `--use-postprocessor NAME`.
|
||||
Plugins can be of `<type>`s `extractor` or `postprocessor`.
|
||||
- Extractor plugins do not need to be enabled from the CLI and are automatically invoked when the input URL is suitable for it.
|
||||
- Extractor plugins take priority over builtin extractors.
|
||||
- Postprocessor plugins can be invoked using `--use-postprocessor NAME`.
|
||||
|
||||
See [ytdlp_plugins](ytdlp_plugins) for example plugins.
|
||||
|
||||
Note that **all** plugins are imported even if not invoked, and that **there are no checks** performed on plugin code. Use plugins at your own risk and only if you trust the code
|
||||
Plugins are loaded from the namespace packages `yt_dlp_plugins.extractor` and `yt_dlp_plugins.postprocessor`.
|
||||
|
||||
If you are a plugin author, add [ytdlp-plugins](https://github.com/topics/ytdlp-plugins) as a topic to your repository for discoverability
|
||||
In other words, the file structure on the disk looks something like:
|
||||
|
||||
yt_dlp_plugins/
|
||||
extractor/
|
||||
myplugin.py
|
||||
postprocessor/
|
||||
myplugin.py
|
||||
|
||||
yt-dlp looks for these `yt_dlp_plugins` namespace folders in many locations (see below) and loads in plugins from **all** of them.
|
||||
|
||||
See the [wiki for some known plugins](https://github.com/yt-dlp/yt-dlp/wiki/Plugins)
|
||||
|
||||
## Installing Plugins
|
||||
|
||||
Plugins can be installed using various methods and locations.
|
||||
|
||||
1. **Configuration directories**:
|
||||
Plugin packages (containing a `yt_dlp_plugins` namespace folder) can be dropped into the following standard [configuration locations](#configuration):
|
||||
* **User Plugins**
|
||||
* `${XDG_CONFIG_HOME}/yt-dlp/plugins/<package name>/yt_dlp_plugins/` (recommended on Linux/macOS)
|
||||
* `${XDG_CONFIG_HOME}/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
|
||||
* `${APPDATA}/yt-dlp/plugins/<package name>/yt_dlp_plugins/` (recommended on Windows)
|
||||
* `~/.yt-dlp/plugins/<package name>/yt_dlp_plugins/`
|
||||
* `~/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
|
||||
* **System Plugins**
|
||||
* `/etc/yt-dlp/plugins/<package name>/yt_dlp_plugins/`
|
||||
* `/etc/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
|
||||
2. **Executable location**: Plugin packages can similarly be installed in a `yt-dlp-plugins` directory under the executable location:
|
||||
* Binary: where `<root-dir>/yt-dlp.exe`, `<root-dir>/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
|
||||
* Source: where `<root-dir>/yt_dlp/__main__.py`, `<root-dir>/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
|
||||
|
||||
3. **pip and other locations in `PYTHONPATH`**
|
||||
* Plugin packages can be installed and managed using `pip`. See [ytdlp-sample-plugins](https://github.com/yt-dlp/yt-dlp-sample-plugins) for an example.
|
||||
* Note: plugin files between plugin packages installed with pip must have unique filenames
|
||||
* Any path in `PYTHONPATH` is searched in for the `yt_dlp_plugins` namespace folder.
|
||||
* Note: This does not apply for Pyinstaller/py2exe builds.
|
||||
|
||||
|
||||
.zip, .egg and .whl archives containing a `yt_dlp_plugins` namespace folder in their root are also supported. These can be placed in the same locations `yt_dlp_plugins` namespace folders can be found.
|
||||
- e.g. `${XDG_CONFIG_HOME}/yt-dlp/plugins/mypluginpkg.zip` where `mypluginpkg.zip` contains `yt_dlp_plugins/<type>/myplugin.py`
|
||||
|
||||
Run yt-dlp with `--verbose`/`-v` to check if the plugin has been loaded.
|
||||
|
||||
## Developing Plugins
|
||||
|
||||
See [ytdlp-sample-plugins](https://github.com/yt-dlp/yt-dlp-sample-plugins) for a sample plugin package with instructions on how to set up an environment for plugin development.
|
||||
|
||||
All public classes with a name ending in `IE` are imported from each file. This respects underscore prefix (e.g. `_MyBasePluginIE` is private) and `__all__`. Modules can similarly be excluded by prefixing the module name with an underscore (e.g. `_myplugin.py`)
|
||||
|
||||
If you are a plugin author, add [yt-dlp-plugins](https://github.com/topics/yt-dlp-plugins) as a topic to your repository for discoverability
|
||||
|
||||
See the [Developer Instructions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) on how to write and test an extractor.
|
||||
|
||||
# EMBEDDING YT-DLP
|
||||
|
||||
|
@ -40,8 +40,12 @@ def main():
|
||||
|
||||
_ALL_CLASSES = get_all_ies() # Must be before import
|
||||
|
||||
import yt_dlp.plugins
|
||||
from yt_dlp.extractor.common import InfoExtractor, SearchInfoExtractor
|
||||
|
||||
# Filter out plugins
|
||||
_ALL_CLASSES = [cls for cls in _ALL_CLASSES if not cls.__module__.startswith(f'{yt_dlp.plugins.PACKAGE_NAME}.')]
|
||||
|
||||
DummyInfoExtractor = type('InfoExtractor', (InfoExtractor,), {'IE_NAME': NO_ATTR})
|
||||
module_src = '\n'.join((
|
||||
MODULE_TEMPLATE,
|
||||
|
73
test/test_plugins.py
Normal file
73
test/test_plugins.py
Normal file
@ -0,0 +1,73 @@
|
||||
import importlib
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
TEST_DATA_DIR = Path(os.path.dirname(os.path.abspath(__file__)), 'testdata')
|
||||
sys.path.append(str(TEST_DATA_DIR))
|
||||
importlib.invalidate_caches()
|
||||
|
||||
from yt_dlp.plugins import PACKAGE_NAME, directories, load_plugins
|
||||
|
||||
|
||||
class TestPlugins(unittest.TestCase):
|
||||
|
||||
TEST_PLUGIN_DIR = TEST_DATA_DIR / PACKAGE_NAME
|
||||
|
||||
def test_directories_containing_plugins(self):
|
||||
self.assertIn(self.TEST_PLUGIN_DIR, map(Path, directories()))
|
||||
|
||||
def test_extractor_classes(self):
|
||||
for module_name in tuple(sys.modules):
|
||||
if module_name.startswith(f'{PACKAGE_NAME}.extractor'):
|
||||
del sys.modules[module_name]
|
||||
plugins_ie = load_plugins('extractor', 'IE')
|
||||
|
||||
self.assertIn(f'{PACKAGE_NAME}.extractor.normal', sys.modules.keys())
|
||||
self.assertIn('NormalPluginIE', plugins_ie.keys())
|
||||
|
||||
# don't load modules with underscore prefix
|
||||
self.assertFalse(
|
||||
f'{PACKAGE_NAME}.extractor._ignore' in sys.modules.keys(),
|
||||
'loaded module beginning with underscore')
|
||||
self.assertNotIn('IgnorePluginIE', plugins_ie.keys())
|
||||
|
||||
# Don't load extractors with underscore prefix
|
||||
self.assertNotIn('_IgnoreUnderscorePluginIE', plugins_ie.keys())
|
||||
|
||||
# Don't load extractors not specified in __all__ (if supplied)
|
||||
self.assertNotIn('IgnoreNotInAllPluginIE', plugins_ie.keys())
|
||||
self.assertIn('InAllPluginIE', plugins_ie.keys())
|
||||
|
||||
def test_postprocessor_classes(self):
|
||||
plugins_pp = load_plugins('postprocessor', 'PP')
|
||||
self.assertIn('NormalPluginPP', plugins_pp.keys())
|
||||
|
||||
def test_importing_zipped_module(self):
|
||||
zip_path = TEST_DATA_DIR / 'zipped_plugins.zip'
|
||||
shutil.make_archive(str(zip_path)[:-4], 'zip', str(zip_path)[:-4])
|
||||
sys.path.append(str(zip_path)) # add zip to search paths
|
||||
importlib.invalidate_caches() # reset the import caches
|
||||
|
||||
try:
|
||||
for plugin_type in ('extractor', 'postprocessor'):
|
||||
package = importlib.import_module(f'{PACKAGE_NAME}.{plugin_type}')
|
||||
self.assertIn(zip_path / PACKAGE_NAME / plugin_type, map(Path, package.__path__))
|
||||
|
||||
plugins_ie = load_plugins('extractor', 'IE')
|
||||
self.assertIn('ZippedPluginIE', plugins_ie.keys())
|
||||
|
||||
plugins_pp = load_plugins('postprocessor', 'PP')
|
||||
self.assertIn('ZippedPluginPP', plugins_pp.keys())
|
||||
|
||||
finally:
|
||||
sys.path.remove(str(zip_path))
|
||||
os.remove(zip_path)
|
||||
importlib.invalidate_caches() # reset the import caches
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
5
test/testdata/yt_dlp_plugins/extractor/_ignore.py
vendored
Normal file
5
test/testdata/yt_dlp_plugins/extractor/_ignore.py
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
from yt_dlp.extractor.common import InfoExtractor
|
||||
|
||||
|
||||
class IgnorePluginIE(InfoExtractor):
|
||||
pass
|
12
test/testdata/yt_dlp_plugins/extractor/ignore.py
vendored
Normal file
12
test/testdata/yt_dlp_plugins/extractor/ignore.py
vendored
Normal file
@ -0,0 +1,12 @@
|
||||
from yt_dlp.extractor.common import InfoExtractor
|
||||
|
||||
|
||||
class IgnoreNotInAllPluginIE(InfoExtractor):
|
||||
pass
|
||||
|
||||
|
||||
class InAllPluginIE(InfoExtractor):
|
||||
pass
|
||||
|
||||
|
||||
__all__ = ['InAllPluginIE']
|
9
test/testdata/yt_dlp_plugins/extractor/normal.py
vendored
Normal file
9
test/testdata/yt_dlp_plugins/extractor/normal.py
vendored
Normal file
@ -0,0 +1,9 @@
|
||||
from yt_dlp.extractor.common import InfoExtractor
|
||||
|
||||
|
||||
class NormalPluginIE(InfoExtractor):
|
||||
pass
|
||||
|
||||
|
||||
class _IgnoreUnderscorePluginIE(InfoExtractor):
|
||||
pass
|
5
test/testdata/yt_dlp_plugins/postprocessor/normal.py
vendored
Normal file
5
test/testdata/yt_dlp_plugins/postprocessor/normal.py
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
|
||||
|
||||
class NormalPluginPP(PostProcessor):
|
||||
pass
|
5
test/testdata/zipped_plugins/yt_dlp_plugins/extractor/zipped.py
vendored
Normal file
5
test/testdata/zipped_plugins/yt_dlp_plugins/extractor/zipped.py
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
from yt_dlp.extractor.common import InfoExtractor
|
||||
|
||||
|
||||
class ZippedPluginIE(InfoExtractor):
|
||||
pass
|
5
test/testdata/zipped_plugins/yt_dlp_plugins/postprocessor/zipped.py
vendored
Normal file
5
test/testdata/zipped_plugins/yt_dlp_plugins/postprocessor/zipped.py
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
|
||||
|
||||
class ZippedPluginPP(PostProcessor):
|
||||
pass
|
@ -32,6 +32,7 @@
|
||||
from .extractor.common import UnsupportedURLIE
|
||||
from .extractor.openload import PhantomJSwrapper
|
||||
from .minicurses import format_text
|
||||
from .plugins import directories as plugin_directories
|
||||
from .postprocessor import _PLUGIN_CLASSES as plugin_postprocessors
|
||||
from .postprocessor import (
|
||||
EmbedThumbnailPP,
|
||||
@ -3773,10 +3774,6 @@ def get_encoding(stream):
|
||||
write_debug('Lazy loading extractors is forcibly disabled')
|
||||
else:
|
||||
write_debug('Lazy loading extractors is disabled')
|
||||
if plugin_extractors or plugin_postprocessors:
|
||||
write_debug('Plugins: %s' % [
|
||||
'%s%s' % (klass.__name__, '' if klass.__name__ == name else f' as {name}')
|
||||
for name, klass in itertools.chain(plugin_extractors.items(), plugin_postprocessors.items())])
|
||||
if self.params['compat_opts']:
|
||||
write_debug('Compatibility options: %s' % ', '.join(self.params['compat_opts']))
|
||||
|
||||
@ -3810,6 +3807,16 @@ def get_encoding(stream):
|
||||
proxy_map.update(handler.proxies)
|
||||
write_debug(f'Proxy map: {proxy_map}')
|
||||
|
||||
for plugin_type, plugins in {'Extractor': plugin_extractors, 'Post-Processor': plugin_postprocessors}.items():
|
||||
if not plugins:
|
||||
continue
|
||||
write_debug(f'{plugin_type} Plugins: %s' % (', '.join(sorted(('%s%s' % (
|
||||
klass.__name__, '' if klass.__name__ == name else f' as {name}')
|
||||
for name, klass in plugins.items())))))
|
||||
plugin_dirs = plugin_directories()
|
||||
if plugin_dirs:
|
||||
write_debug(f'Plugin directories: {plugin_dirs}')
|
||||
|
||||
# Not implemented
|
||||
if False and self.params.get('call_home'):
|
||||
ipaddr = self.urlopen('https://yt-dl.org/ip').read().decode()
|
||||
|
@ -1,10 +1,10 @@
|
||||
import contextlib
|
||||
import os
|
||||
|
||||
from ..utils import load_plugins
|
||||
from ..plugins import load_plugins
|
||||
|
||||
# NB: Must be before other imports so that plugins can be correctly injected
|
||||
_PLUGIN_CLASSES = load_plugins('extractor', 'IE', {})
|
||||
_PLUGIN_CLASSES = load_plugins('extractor', 'IE')
|
||||
|
||||
_LAZY_LOADER = False
|
||||
if not os.environ.get('YTDLP_NO_LAZY_EXTRACTORS'):
|
||||
|
@ -29,6 +29,8 @@
|
||||
expand_path,
|
||||
format_field,
|
||||
get_executable_path,
|
||||
get_system_config_dirs,
|
||||
get_user_config_dirs,
|
||||
join_nonempty,
|
||||
orderedSet_from_options,
|
||||
remove_end,
|
||||
@ -42,50 +44,55 @@ def parseOpts(overrideArguments=None, ignore_config_files='if_override'):
|
||||
if ignore_config_files == 'if_override':
|
||||
ignore_config_files = overrideArguments is not None
|
||||
|
||||
def _readUserConf(package_name, default=[]):
|
||||
# .config
|
||||
def _load_from_config_dirs(config_dirs):
|
||||
for config_dir in config_dirs:
|
||||
conf_file_path = os.path.join(config_dir, 'config')
|
||||
conf = Config.read_file(conf_file_path, default=None)
|
||||
if conf is None:
|
||||
conf_file_path += '.txt'
|
||||
conf = Config.read_file(conf_file_path, default=None)
|
||||
if conf is not None:
|
||||
return conf, conf_file_path
|
||||
return None, None
|
||||
|
||||
def _read_user_conf(package_name, default=None):
|
||||
# .config/package_name.conf
|
||||
xdg_config_home = os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config')
|
||||
userConfFile = os.path.join(xdg_config_home, package_name, 'config')
|
||||
if not os.path.isfile(userConfFile):
|
||||
userConfFile = os.path.join(xdg_config_home, '%s.conf' % package_name)
|
||||
userConf = Config.read_file(userConfFile, default=None)
|
||||
if userConf is not None:
|
||||
return userConf, userConfFile
|
||||
user_conf_file = os.path.join(xdg_config_home, '%s.conf' % package_name)
|
||||
user_conf = Config.read_file(user_conf_file, default=None)
|
||||
if user_conf is not None:
|
||||
return user_conf, user_conf_file
|
||||
|
||||
# appdata
|
||||
appdata_dir = os.getenv('appdata')
|
||||
if appdata_dir:
|
||||
userConfFile = os.path.join(appdata_dir, package_name, 'config')
|
||||
userConf = Config.read_file(userConfFile, default=None)
|
||||
if userConf is None:
|
||||
userConfFile += '.txt'
|
||||
userConf = Config.read_file(userConfFile, default=None)
|
||||
if userConf is not None:
|
||||
return userConf, userConfFile
|
||||
# home (~/package_name.conf or ~/package_name.conf.txt)
|
||||
user_conf_file = os.path.join(compat_expanduser('~'), '%s.conf' % package_name)
|
||||
user_conf = Config.read_file(user_conf_file, default=None)
|
||||
if user_conf is None:
|
||||
user_conf_file += '.txt'
|
||||
user_conf = Config.read_file(user_conf_file, default=None)
|
||||
if user_conf is not None:
|
||||
return user_conf, user_conf_file
|
||||
|
||||
# home
|
||||
userConfFile = os.path.join(compat_expanduser('~'), '%s.conf' % package_name)
|
||||
userConf = Config.read_file(userConfFile, default=None)
|
||||
if userConf is None:
|
||||
userConfFile += '.txt'
|
||||
userConf = Config.read_file(userConfFile, default=None)
|
||||
if userConf is not None:
|
||||
return userConf, userConfFile
|
||||
# Package config directories (e.g. ~/.config/package_name/package_name.txt)
|
||||
user_conf, user_conf_file = _load_from_config_dirs(get_user_config_dirs(package_name))
|
||||
if user_conf is not None:
|
||||
return user_conf, user_conf_file
|
||||
return default if default is not None else [], None
|
||||
|
||||
return default, None
|
||||
def _read_system_conf(package_name, default=None):
|
||||
system_conf, system_conf_file = _load_from_config_dirs(get_system_config_dirs(package_name))
|
||||
if system_conf is not None:
|
||||
return system_conf, system_conf_file
|
||||
return default if default is not None else [], None
|
||||
|
||||
def add_config(label, path, user=False):
|
||||
def add_config(label, path=None, func=None):
|
||||
""" Adds config and returns whether to continue """
|
||||
if root.parse_known_args()[0].ignoreconfig:
|
||||
return False
|
||||
# Multiple package names can be given here
|
||||
# E.g. ('yt-dlp', 'youtube-dlc', 'youtube-dl') will look for
|
||||
# the configuration file of any of these three packages
|
||||
for package in ('yt-dlp',):
|
||||
if user:
|
||||
args, current_path = _readUserConf(package, default=None)
|
||||
elif func:
|
||||
assert path is None
|
||||
args, current_path = func('yt-dlp')
|
||||
else:
|
||||
current_path = os.path.join(path, '%s.conf' % package)
|
||||
current_path = os.path.join(path, 'yt-dlp.conf')
|
||||
args = Config.read_file(current_path, default=None)
|
||||
if args is not None:
|
||||
root.append_config(args, current_path, label=label)
|
||||
@ -96,8 +103,8 @@ def load_configs():
|
||||
yield not ignore_config_files
|
||||
yield add_config('Portable', get_executable_path())
|
||||
yield add_config('Home', expand_path(root.parse_known_args()[0].paths.get('home', '')).strip())
|
||||
yield add_config('User', None, user=True)
|
||||
yield add_config('System', '/etc')
|
||||
yield add_config('User', func=_read_user_conf)
|
||||
yield add_config('System', func=_read_system_conf)
|
||||
|
||||
opts = optparse.Values({'verbose': True, 'print_help': False})
|
||||
try:
|
||||
|
171
yt_dlp/plugins.py
Normal file
171
yt_dlp/plugins.py
Normal file
@ -0,0 +1,171 @@
|
||||
import contextlib
|
||||
import importlib
|
||||
import importlib.abc
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import inspect
|
||||
import itertools
|
||||
import os
|
||||
import pkgutil
|
||||
import sys
|
||||
import traceback
|
||||
import zipimport
|
||||
from pathlib import Path
|
||||
from zipfile import ZipFile
|
||||
|
||||
from .compat import functools # isort: split
|
||||
from .compat import compat_expanduser
|
||||
from .utils import (
|
||||
get_executable_path,
|
||||
get_system_config_dirs,
|
||||
get_user_config_dirs,
|
||||
write_string,
|
||||
)
|
||||
|
||||
PACKAGE_NAME = 'yt_dlp_plugins'
|
||||
COMPAT_PACKAGE_NAME = 'ytdlp_plugins'
|
||||
|
||||
|
||||
class PluginLoader(importlib.abc.Loader):
|
||||
"""Dummy loader for virtual namespace packages"""
|
||||
|
||||
def exec_module(self, module):
|
||||
return None
|
||||
|
||||
|
||||
@functools.cache
|
||||
def dirs_in_zip(archive):
|
||||
with ZipFile(archive) as zip:
|
||||
return set(itertools.chain.from_iterable(
|
||||
Path(file).parents for file in zip.namelist()))
|
||||
|
||||
|
||||
class PluginFinder(importlib.abc.MetaPathFinder):
|
||||
"""
|
||||
This class provides one or multiple namespace packages.
|
||||
It searches in sys.path and yt-dlp config folders for
|
||||
the existing subdirectories from which the modules can be imported
|
||||
"""
|
||||
|
||||
def __init__(self, *packages):
|
||||
self._zip_content_cache = {}
|
||||
self.packages = set(itertools.chain.from_iterable(
|
||||
itertools.accumulate(name.split('.'), lambda a, b: '.'.join((a, b)))
|
||||
for name in packages))
|
||||
|
||||
def search_locations(self, fullname):
|
||||
candidate_locations = []
|
||||
|
||||
def _get_package_paths(*root_paths, containing_folder='plugins'):
|
||||
for config_dir in map(Path, root_paths):
|
||||
plugin_dir = config_dir / containing_folder
|
||||
if not plugin_dir.is_dir():
|
||||
continue
|
||||
yield from plugin_dir.iterdir()
|
||||
|
||||
# Load from yt-dlp config folders
|
||||
candidate_locations.extend(_get_package_paths(
|
||||
*get_user_config_dirs('yt-dlp'), *get_system_config_dirs('yt-dlp'),
|
||||
containing_folder='plugins'))
|
||||
|
||||
# Load from yt-dlp-plugins folders
|
||||
candidate_locations.extend(_get_package_paths(
|
||||
get_executable_path(),
|
||||
compat_expanduser('~'),
|
||||
'/etc',
|
||||
os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config'),
|
||||
containing_folder='yt-dlp-plugins'))
|
||||
|
||||
candidate_locations.extend(map(Path, sys.path)) # PYTHONPATH
|
||||
|
||||
parts = Path(*fullname.split('.'))
|
||||
locations = set()
|
||||
for path in dict.fromkeys(candidate_locations):
|
||||
candidate = path / parts
|
||||
if candidate.is_dir():
|
||||
locations.add(str(candidate))
|
||||
elif path.name and any(path.with_suffix(suffix).is_file() for suffix in {'.zip', '.egg', '.whl'}):
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
if parts in dirs_in_zip(path):
|
||||
locations.add(str(candidate))
|
||||
return locations
|
||||
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname not in self.packages:
|
||||
return None
|
||||
|
||||
search_locations = self.search_locations(fullname)
|
||||
if not search_locations:
|
||||
return None
|
||||
|
||||
spec = importlib.machinery.ModuleSpec(fullname, PluginLoader(), is_package=True)
|
||||
spec.submodule_search_locations = search_locations
|
||||
return spec
|
||||
|
||||
def invalidate_caches(self):
|
||||
dirs_in_zip.cache_clear()
|
||||
for package in self.packages:
|
||||
if package in sys.modules:
|
||||
del sys.modules[package]
|
||||
|
||||
|
||||
def directories():
|
||||
spec = importlib.util.find_spec(PACKAGE_NAME)
|
||||
return spec.submodule_search_locations if spec else []
|
||||
|
||||
|
||||
def iter_modules(subpackage):
|
||||
fullname = f'{PACKAGE_NAME}.{subpackage}'
|
||||
with contextlib.suppress(ModuleNotFoundError):
|
||||
pkg = importlib.import_module(fullname)
|
||||
yield from pkgutil.iter_modules(path=pkg.__path__, prefix=f'{fullname}.')
|
||||
|
||||
|
||||
def load_module(module, module_name, suffix):
|
||||
return inspect.getmembers(module, lambda obj: (
|
||||
inspect.isclass(obj)
|
||||
and obj.__name__.endswith(suffix)
|
||||
and obj.__module__.startswith(module_name)
|
||||
and not obj.__name__.startswith('_')
|
||||
and obj.__name__ in getattr(module, '__all__', [obj.__name__])))
|
||||
|
||||
|
||||
def load_plugins(name, suffix):
|
||||
classes = {}
|
||||
|
||||
for finder, module_name, _ in iter_modules(name):
|
||||
if any(x.startswith('_') for x in module_name.split('.')):
|
||||
continue
|
||||
try:
|
||||
if sys.version_info < (3, 10) and isinstance(finder, zipimport.zipimporter):
|
||||
# zipimporter.load_module() is deprecated in 3.10 and removed in 3.12
|
||||
# The exec_module branch below is the replacement for >= 3.10
|
||||
# See: https://docs.python.org/3/library/zipimport.html#zipimport.zipimporter.exec_module
|
||||
module = finder.load_module(module_name)
|
||||
else:
|
||||
spec = finder.find_spec(module_name)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[module_name] = module
|
||||
spec.loader.exec_module(module)
|
||||
except Exception:
|
||||
write_string(f'Error while importing module {module_name!r}\n{traceback.format_exc(limit=-1)}')
|
||||
continue
|
||||
classes.update(load_module(module, module_name, suffix))
|
||||
|
||||
# Compat: old plugin system using __init__.py
|
||||
# Note: plugins imported this way do not show up in directories()
|
||||
# nor are considered part of the yt_dlp_plugins namespace package
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
name, Path(get_executable_path(), COMPAT_PACKAGE_NAME, name, '__init__.py'))
|
||||
plugins = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = plugins
|
||||
spec.loader.exec_module(plugins)
|
||||
classes.update(load_module(plugins, spec.name, suffix))
|
||||
|
||||
return classes
|
||||
|
||||
|
||||
sys.meta_path.insert(0, PluginFinder(f'{PACKAGE_NAME}.extractor', f'{PACKAGE_NAME}.postprocessor'))
|
||||
|
||||
__all__ = ['directories', 'load_plugins', 'PACKAGE_NAME', 'COMPAT_PACKAGE_NAME']
|
@ -33,14 +33,15 @@
|
||||
from .sponskrub import SponSkrubPP
|
||||
from .sponsorblock import SponsorBlockPP
|
||||
from .xattrpp import XAttrMetadataPP
|
||||
from ..utils import load_plugins
|
||||
from ..plugins import load_plugins
|
||||
|
||||
_PLUGIN_CLASSES = load_plugins('postprocessor', 'PP', globals())
|
||||
_PLUGIN_CLASSES = load_plugins('postprocessor', 'PP')
|
||||
|
||||
|
||||
def get_postprocessor(key):
|
||||
return globals()[key + 'PP']
|
||||
|
||||
|
||||
globals().update(_PLUGIN_CLASSES)
|
||||
__all__ = [name for name in globals().keys() if name.endswith('PP')]
|
||||
__all__.extend(('PostProcessor', 'FFmpegPostProcessor'))
|
||||
|
@ -18,7 +18,6 @@
|
||||
import html.parser
|
||||
import http.client
|
||||
import http.cookiejar
|
||||
import importlib.util
|
||||
import inspect
|
||||
import io
|
||||
import itertools
|
||||
@ -5372,22 +5371,37 @@ def get_executable_path():
|
||||
return os.path.dirname(os.path.abspath(_get_variant_and_executable_path()[1]))
|
||||
|
||||
|
||||
def load_plugins(name, suffix, namespace):
|
||||
classes = {}
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
plugins_spec = importlib.util.spec_from_file_location(
|
||||
name, os.path.join(get_executable_path(), 'ytdlp_plugins', name, '__init__.py'))
|
||||
plugins = importlib.util.module_from_spec(plugins_spec)
|
||||
sys.modules[plugins_spec.name] = plugins
|
||||
plugins_spec.loader.exec_module(plugins)
|
||||
for name in dir(plugins):
|
||||
if name in namespace:
|
||||
continue
|
||||
if not name.endswith(suffix):
|
||||
continue
|
||||
klass = getattr(plugins, name)
|
||||
classes[name] = namespace[name] = klass
|
||||
return classes
|
||||
def get_user_config_dirs(package_name):
|
||||
locations = set()
|
||||
|
||||
# .config (e.g. ~/.config/package_name)
|
||||
xdg_config_home = os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config')
|
||||
config_dir = os.path.join(xdg_config_home, package_name)
|
||||
if os.path.isdir(config_dir):
|
||||
locations.add(config_dir)
|
||||
|
||||
# appdata (%APPDATA%/package_name)
|
||||
appdata_dir = os.getenv('appdata')
|
||||
if appdata_dir:
|
||||
config_dir = os.path.join(appdata_dir, package_name)
|
||||
if os.path.isdir(config_dir):
|
||||
locations.add(config_dir)
|
||||
|
||||
# home (~/.package_name)
|
||||
user_config_directory = os.path.join(compat_expanduser('~'), '.%s' % package_name)
|
||||
if os.path.isdir(user_config_directory):
|
||||
locations.add(user_config_directory)
|
||||
|
||||
return locations
|
||||
|
||||
|
||||
def get_system_config_dirs(package_name):
|
||||
locations = set()
|
||||
# /etc/package_name
|
||||
system_config_directory = os.path.join('/etc', package_name)
|
||||
if os.path.isdir(system_config_directory):
|
||||
locations.add(system_config_directory)
|
||||
return locations
|
||||
|
||||
|
||||
def traverse_obj(
|
||||
@ -6367,3 +6381,10 @@ def calculate_preference(self, format):
|
||||
# Deprecated
|
||||
has_certifi = bool(certifi)
|
||||
has_websockets = bool(websockets)
|
||||
|
||||
|
||||
def load_plugins(name, suffix, namespace):
|
||||
from .plugins import load_plugins
|
||||
ret = load_plugins(name, suffix)
|
||||
namespace.update(ret)
|
||||
return ret
|
||||
|
@ -1,4 +0,0 @@
|
||||
# flake8: noqa: F401
|
||||
|
||||
# ℹ️ The imported name must end in "IE"
|
||||
from .sample import SamplePluginIE
|
@ -1,14 +0,0 @@
|
||||
# ⚠ Don't use relative imports
|
||||
from yt_dlp.extractor.common import InfoExtractor
|
||||
|
||||
|
||||
# ℹ️ Instructions on making extractors can be found at:
|
||||
# 🔗 https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-support-for-a-new-site
|
||||
|
||||
class SamplePluginIE(InfoExtractor):
|
||||
_WORKING = False
|
||||
IE_DESC = False
|
||||
_VALID_URL = r'^sampleplugin:'
|
||||
|
||||
def _real_extract(self, url):
|
||||
self.to_screen('URL "%s" successfully captured' % url)
|
@ -1,4 +0,0 @@
|
||||
# flake8: noqa: F401
|
||||
|
||||
# ℹ️ The imported name must end in "PP" and is the name to be used in --use-postprocessor
|
||||
from .sample import SamplePluginPP
|
@ -1,26 +0,0 @@
|
||||
# ⚠ Don't use relative imports
|
||||
from yt_dlp.postprocessor.common import PostProcessor
|
||||
|
||||
|
||||
# ℹ️ See the docstring of yt_dlp.postprocessor.common.PostProcessor
|
||||
class SamplePluginPP(PostProcessor):
|
||||
def __init__(self, downloader=None, **kwargs):
|
||||
# ⚠ Only kwargs can be passed from the CLI, and all argument values will be string
|
||||
# Also, "downloader", "when" and "key" are reserved names
|
||||
super().__init__(downloader)
|
||||
self._kwargs = kwargs
|
||||
|
||||
# ℹ️ See docstring of yt_dlp.postprocessor.common.PostProcessor.run
|
||||
def run(self, info):
|
||||
if info.get('_type', 'video') != 'video': # PP was called for playlist
|
||||
self.to_screen(f'Post-processing playlist {info.get("id")!r} with {self._kwargs}')
|
||||
elif info.get('filepath'): # PP was called after download (default)
|
||||
filepath = info.get('filepath')
|
||||
self.to_screen(f'Post-processed {filepath!r} with {self._kwargs}')
|
||||
elif info.get('requested_downloads'): # PP was called after_video
|
||||
filepaths = [f.get('filepath') for f in info.get('requested_downloads')]
|
||||
self.to_screen(f'Post-processed {filepaths!r} with {self._kwargs}')
|
||||
else: # PP was called before actual download
|
||||
filepath = info.get('_filename')
|
||||
self.to_screen(f'Pre-processed {filepath!r} with {self._kwargs}')
|
||||
return [], info # return list_of_files_to_delete, info_dict
|
Loading…
Reference in New Issue
Block a user