Commit Graph

169 Commits

Author SHA1 Message Date
chris
f522573787
[extractor] Extract chapters from JSON-LD (#2031)
Authored by: iw0nderhow, pukkandan
2022-01-02 02:37:00 +05:30
nixxo
7592749cbe
[extractor] Extract thumbnails from JSON-LD (#2195)
Authored by: nixxo
2022-01-02 01:20:27 +05:30
Felix S
4afa3ec4b6
[extractor] Detect more subtitle codecs in MPD manifests (#2174)
Authored by: fstirlitz
2022-01-01 01:36:45 +05:30
MinePlayersPE
d43de6821c
[GameJolt] Add extractors (#2036)
Authored by: MinePlayersPE
2021-12-25 03:58:57 +05:30
siddharth
774a46c53d
[npr] Make SMIL extraction non-fatal (#2099)
Closes #1934
Authored by: r5d
2021-12-24 07:45:48 +05:30
pukkandan
b69fd25c25
[cleanup] Misc cleanup
Closes #1942 #1976 #2020 #2058 #1984
2021-12-23 07:12:46 +05:30
The Hatsune Daishi
adbc4ec4bb
[dash,youtube] Download live from start to end (#888)
* Add option `--live-from-start` to enable downloading live videos from start
* Add key `is_from_start` in formats to identify formats (of live videos) that downloads from start
* [dash] Create protocol `http_dash_segments_generator` that allows a function to be passed instead of fragments
* [fragment] Allow multiple live dash formats to download simultaneously
* [youtube] Implement fragment re-fetching for the live dash formats
* [youtube] Re-extract dash manifest every 5 hours (manifest expires in 6hrs)
* [postprocessor/ffmpeg] Add `FFmpegFixupDuplicateMoovPP` to fixup duplicated moov atoms

Known issue: Ctrl+C doesn't work on Windows when downloading multiple formats

Closes #1521
Authored by: nao20010128nao, pukkandan
2021-12-20 11:36:46 +05:30
Zenon Mousmoulas
d5c3254889
[extractor] Support default implicit graph in JSON-LD (#1983)
Original PR: https://github.com/ytdl-org/youtube-dl/pull/30229

Per W3C JSON-LD v1.1 §4.9 (non-normative ref):

    When a JSON-LD document's top-level structure is a map that contains
    no other keys than @graph and optionally @context (properties that
    are not mapped to an IRI or a keyword are ignored), @graph is
    considered to express the otherwise implicit default graph.

Authored by: zmousm
2021-12-17 02:16:30 +05:30
pukkandan
a8549f19e7
[tiktok] Fix 53dad39e30 2021-12-15 22:18:01 +05:30
pukkandan
39ca3b5c7f
[extractor] Standardize _live_title 2021-12-15 22:09:07 +05:30
pukkandan
19188702ef
[FormatSort] Prevent incorrect deprecation warning
Closes #1981
2021-12-15 01:33:15 +05:30
The Hatsune Daishi
66f4c04e50
[extractor] Add _search_nuxt_data (#1921)
Authored by: nao20010128nao
2021-12-07 22:08:50 +05:30
pukkandan
d2b2fca53f
[extractor] Ignore errors in comment extraction when -i is given
Closes #1787
2021-12-03 03:46:04 +05:30
pukkandan
2c4aaaddc9
[lazy_extractors] Fix for search IEs
Closes #1851
2021-12-01 23:23:59 +05:30
pukkandan
ee8dd27a73
[cleanup] Add deprecation warnings 2021-11-29 23:34:33 +05:30
pukkandan
f304da8a29
[cleanup] Misc cleanup
Closes #1805, closes #1800
2021-11-29 23:34:33 +05:30
pukkandan
f2ebc5c7be
Option --wait-for-video to wait for scheduled streams 2021-11-29 22:52:01 +05:30
pukkandan
8f122fa070
[extractor] Extract average_rating from JSON-LD
Eg: Crunchyroll
2021-11-23 13:14:06 +05:30
pukkandan
234416e4bf
[downloader/ffmpeg] Fix for direct videos inside mpd manifests
Closes #1751
2021-11-23 01:29:10 +05:30
nyuszika7h
6b993ca765
[hls] Better FairPlay DRM detection (#1661)
Authored by: nyuszika7h
2021-11-19 07:19:51 +05:30
pukkandan
9ac24e235e
[curiositystream] Add more metadata
Closes #1568
2021-11-13 23:49:14 +05:30
Lauren Liberda
f98709af31
[extractor] Add _search_nextjs_data (#1386)
Authored by: selfisekai
2021-11-10 06:11:05 +05:30
pukkandan
34921b4345
[utils] Add join_nonempty 2021-11-06 07:53:55 +05:30
pukkandan
aeb2a9ad27
[FormatSort] eac3 is better than ac3 2021-11-05 20:40:45 +05:30
pukkandan
0930b11fda
[docs,cleanup] Improve docs and minor cleanup
Closes #1387, #1404, #1408, #1485, #1415, #1450, #1492
2021-10-31 14:47:33 +05:30
pukkandan
10beccc980
[FormatSort] Fix some fields' defaults
Closes #1479
2021-10-30 01:14:14 +05:30
pukkandan
0db3bae879
[extractor] Fix some errors being converted to ExtractorError 2021-10-26 20:27:09 +05:30
pukkandan
96565c7e55
[cleanup] Add keyword automatically to SearchIE descriptions
and some minor cleanup of docs
2021-10-23 21:20:19 +05:30
pukkandan
ec11a9f4a2
[minicurses] Add more colors 2021-10-23 05:23:38 +05:30
pukkandan
49a57e70a9
[cleanup] misc 2021-10-23 02:09:10 +05:30
pukkandan
176f1866cb
Add HDR information to formats 2021-10-18 18:35:02 +05:30
pukkandan
e820fbaa6f
Do not verify thumbnail URLs by default
Partially reverts cca80fe611 and 0ba692acc8

Unless `--check-formats` is specified, this causes yt-dlp to return incorrect thumbnail urls.
See https://github.com/yt-dlp/yt-dlp/issues/340#issuecomment-877909966, #402

But the overhead in general use is not worth it

Closes #694, #725
2021-10-18 15:44:47 +05:30
pukkandan
ec3f6640c1
[crunchyroll] Add season to flat-playlist
Closes #1319
2021-10-17 17:39:23 +05:30
pukkandan
a0c716bb61
[instagram] Show appropriate error when login is needed
Closes #1264
2021-10-14 14:44:29 +05:30
pukkandan
6993f78d1b
[extractor,utils] Detect more codecs/mimetypes
Fixes: https://github.com/ytdl-org/youtube-dl/issues/29943
2021-10-13 05:05:29 +05:30
pukkandan
a2160aa45f
[extractor] Generalize getcomments implementation 2021-10-12 15:21:30 +05:30
pukkandan
cc16383ff3
[extractor] Simplify search extractors 2021-10-12 15:21:30 +05:30
pukkandan
ba10757412 [extractor] Detect EXT-X-KEY Apple FairPlay 2021-10-11 17:09:21 +05:30
pukkandan
b5ae35ee6d
[cleanup] Misc cleanup 2021-10-09 22:32:00 +05:30
pukkandan
819e05319b
Improved progress reporting (See desc) (#1125)
* Separate `--console-title` and `--no-progress`
* Add option `--progress` to show progress-bar even in quiet mode
* Fix and refactor `minicurses`
* Use `minicurses` for all progress reporting
* Standardize use of terminal sequences and enable color support for windows 10
* Add option `--progress-template` to customize progress-bar and console-title
* Add postprocessor hooks and progress reporting

Closes: #906, #901, #1085, #1170
2021-10-09 00:41:59 +05:30
Felix S
9359f3d4f0
[extractor] Extract storyboards from SMIL manifests (#1128)
Authored by: fstirlitz
2021-10-03 00:13:42 +05:30
pukkandan
e6f21b3d92
[docs,cleanup] Some minor refactoring and improve docs 2021-09-30 03:32:52 +05:30
ChillingPepper
3cf4b91dc5
[SovietsCloset] Add duration from m3u8 (#908)
Authored by: ChillingPepper
2021-09-28 02:30:41 +05:30
pukkandan
1813a6ccd4
[youtube] Fix --mark-watched with --cookies-from-browser
Closes #1019
2021-09-24 05:14:16 +05:30
pukkandan
bd6f722de8
dump files should obey --trim-filename (#1043)
Authored by: sulyi
2021-09-22 05:25:17 +05:30
DigitalDJ
f9cc0161e6
[extractor] Fix root-relative URLs in MPD (#1006)
Authored by: DigitalDJ
2021-09-19 14:07:57 +05:30
pukkandan
0001fcb586
Add option --netrc-location
Closes #792, #963
2021-09-16 01:28:55 +05:30
pukkandan
198e3a04c9
[FormatSort] Remove priority of lang 2021-08-26 08:08:33 +05:30
pukkandan
6251555f1c
[downloader/ffmpeg] Support for DASH manifests (experimental)
Closes #159
2021-08-24 05:52:00 +05:30
pukkandan
1151c4079a
[extractor] Show video id in error messages if possible 2021-08-23 02:49:07 +05:30
pukkandan
88acdbc269
[extractor] Better error message for DRM (#729)
Closes #636
2021-08-23 01:38:38 +05:30
pukkandan
3fb4e21b38
[lazy_extractors] Fix suitable and add flake8 test 2021-08-23 01:04:29 +05:30
pukkandan
5ad28e7ffd
[extractor] Common function _match_valid_url 2021-08-23 00:50:40 +05:30
pukkandan
a8731fcc1d minor bugfixes
bugs due to be2fc5b212, e9f4ccd19e
2021-08-11 20:27:30 +05:30
pukkandan
be2fc5b212
[extractor] Detect sttp as subtitles in MPD
Closes #656
Solution by: fstirlitz
2021-08-10 04:46:48 +05:30
pukkandan
b868936cd6 [cleanup] Misc 2021-08-07 21:17:07 +05:30
pukkandan
28f436bad0 [extractor] Reset non-repeating warnings per video 2021-08-07 21:17:05 +05:30
MinePlayersPE
3dbb2a9dcb
[RCTIPlus] Support events and TV (#625)
Authored by: MinePlayersPE
2021-08-04 18:42:15 +05:30
LE
68f5867cf0
[CBS] Add fallback (#579)
Related: https://github.com/ytdl-org/youtube-dl/issues/29564
Authored-by: llacb47, pukkandan
2021-08-02 07:46:12 +05:30
pukkandan
dbf5416a20 [cleanup] Refactor some code 2021-08-01 12:38:05 +05:30
pukkandan
f5510afef0
[FormatSort] Fix bug for audio with unknown codec 2021-08-01 12:13:40 +05:30
pukkandan
ae30b84072
Add field live_status 2021-07-21 20:50:58 +05:30
pukkandan
0ba692acc8
[youtube] Extract more thumbnails
* The thumbnail URLs are hard-coded and their actual existence is tested lazily
* Added option `--no-check-formats` to not test them

Closes #340, Related: #402, #337, https://github.com/ytdl-org/youtube-dl/issues/29049
2021-07-20 02:42:11 +05:30
Felix S
da1c94ee45
[generic] Extract previously missed subtitles (#515)
* [generic] Extract subtitles in cases missed previously
* [common] Detect discarded subtitles in SMIL manifests
* [generic] Extract everything in the SMIL manifest

Authored by: fstirlitz
2021-07-16 19:52:56 +05:30
felix
198f7ea89e
[extractor] Allow extracting multiple groups in _search_regex
From #497, Authored by: fstirlitz
2021-07-14 05:02:42 +05:30
pukkandan
6606817a86
[utils] Add variadic 2021-07-11 04:46:51 +05:30
pukkandan
4bb6b02f93
Improve extractor_args parsing 2021-07-08 21:22:35 +05:30
pukkandan
46890374f7
[extractor] Minor improvements (See desc)
1. Allow removal of login hint - extractors can set their own login hint as part of `msg`
2. Cleanup `_merge_subtitles` signature
2021-07-07 02:27:53 +05:30
pukkandan
60755938b3
[extractor] Prevent unnecessary download of hls manifests
and refactor `hls_split_discontinuity` code
2021-07-07 02:24:58 +05:30
pukkandan
5d3a0e794b
Add --extractor-args to pass extractor-specific arguments 2021-06-25 20:10:28 +05:30
pukkandan
e36d50c5dd
[websockets] Add WebSocketFragmentFD (#399)
Necessary for #392

Co-authored by: nao20010128nao, pukkandan
2021-06-21 22:56:36 +05:30
pukkandan
d0fb4bd16f
[pornhub] Extract cast
Closes #406, https://github.com/ytdl-org/youtube-dl/pull/27384
2021-06-13 21:38:08 +05:30
felix
cdb19aa4c2
[downloader/mhtml] Add new downloader (#343)
This downloader is intended to be used for streams that consist of a
timed sequence of stand-alone images, such as slideshows or thumbnail
streams

This can be used for implementing:

https://github.com/ytdl-org/youtube-dl/issues/4974#issue-58006762
https://github.com/ytdl-org/youtube-dl/issues/4540#issuecomment-69574231
https://github.com/ytdl-org/youtube-dl/pull/11185#issuecomment-335554239

https://github.com/ytdl-org/youtube-dl/issues/9868
https://github.com/ytdl-org/youtube-dl/pull/14951


Authored by: fstirlitz
2021-06-13 20:36:40 +05:30
pukkandan
8326b00aab
Allow images formats
Necessary for #343.

* They are identified by `vcodec=acodec='none'`
* These formats show as the worst in `-F`
* Any postprocessor that expects audio/video will be skipped
* `b*` and all related selectors will skip such formats
* This commit also does not add any selector for downloading such formats. They have to be explicitly requested by the `format_id`. Implementation of a selector is left for when #389 is resolved
2021-06-13 03:45:53 +05:30
pukkandan
3b1fe47d84
Release 2021.06.08 2021-06-08 20:13:41 +05:30
felix
e2efe599aa
[common] Fix FourCC fallback when parsing ISM (#372)
In some DASH manifests, the FourCC attribute is actually present,
but empty.  We thus apply the same fallback to 'AACL' that we do
when the attribute is entirely absent.

Authored by: fstirlitz
2021-06-06 00:59:04 +05:30
pukkandan
b0089e8992
[fancode] Add extractor (#316,#354)
Closes #269, #363

Authored by: rmsmachine
2021-05-30 19:34:39 +05:30
pukkandan
adddc50cbf
[extractor] Functions to parse socket.io response as json
Authored by: pukkandan, llacb47
2021-05-30 14:22:42 +05:30
pukkandan
37a3bb66a7
[extractor] Allow note=False when extracting manifests 2021-05-29 14:22:44 +05:30
pukkandan
177877c544
[extractor] Always prefer native hls downloader by default
When the manifest is not downloadable by native downloader, it already is able to detect it and switch to `ffmpeg`. So there doesn't seem to be a reason anymore to use ffmpeg as the preferred downloader
2021-05-26 01:27:39 +05:30
pukkandan
3907333c5d
[extractor] Skip subtitles without URI in m3u8 manifests
Closes #339

Authored by: hheimbuerger
2021-05-23 22:32:47 +05:30
pukkandan
9d5d4d64f8
[youtube] Better message when login required 2021-05-20 15:55:55 +05:30
pukkandan
07cce701de
[cleanup] linter, code formatting and readme 2021-05-19 16:48:20 +05:30
pukkandan
a06916d98e
[extractor] Add write_debug and get_param 2021-05-17 18:59:51 +05:30
pukkandan
2412044c90
Add field name for subtitles
Co-authored by: pukkandan, tpikonen

Based on: #310, https://github.com/ytdl-org/youtube-dl/pull/26112
2021-05-15 19:27:52 +05:30
pukkandan
0760b0a7e2
Standardize write_debug 2021-05-14 13:15:29 +05:30
pukkandan
3d89341b47
[common] bugfix for when compat_opts is not given 2021-05-11 23:29:26 +05:30
pukkandan
fac988053f
Release 2021.05.11
* and some documentation improvements
2021-05-11 13:35:05 +05:30
pukkandan
53ed7066ab
Option --compat-options to revert some of yt-dlp's changes
* Deprecates `--list-formats-as-table`, `--list-formats-old`
2021-05-11 13:30:48 +05:30
pukkandan
3158150cb7
[utils] Add network_exceptions 2021-05-04 22:36:18 +05:30
The Hatsune Daishi
0fa9a1e236
[whowatch] Add extractor #292
closes #223

Authored by: nao20010128nao 
Modified from: 9e4a0e061a/youtube_dl/extractor/whowatch.py
2021-05-02 19:43:37 +05:30
Felix S
66a1b8643a [downloader/ism] Support muxing TTML subtitles 2021-04-28 17:21:45 +05:30
Felix S
4a2f19abbd [downloader/hls] Assemble single-file WebVTT subtitles from HLS segments 2021-04-28 17:21:14 +05:30
Felix S
f6a1d69a87 [extractor/common] Extend _extract_akamai_formats to also extract subtitle tracks 2021-04-28 17:20:29 +05:30
Felix S
fd76a14259 [extractor/common, downloader/ism] Extract SSTR subtitle tracks
_parse_ism_formats was extended into _parse_ism_formats_and_subtitles;
all direct users were updated, though _extract_ism_formats was left
as a compatibility wrapper.

The SSTR downloader was also modified in order to prepare for muxing
subtitle streams, although no support for any subtitle codecs was
added in this commit.
2021-04-28 17:20:20 +05:30
Felix S
171e59edd4 [extractor/common] Extract DASH subtitle tracks
_extract_mpd_formats and _parse_mpd_formats were extended into
_…_formats_and_subtitles; wrappers with old names are provided
for compatibility.
2021-04-28 17:20:11 +05:30
Felix S
a0c3b2d5cf [extractor/common] Extract HLS subtitle tracks
_extract_m3u8_formats is renamed to _extract_m3u8_formats_and_subtitles
and extended to handle subtitle tracks instead of skipping them;
a wrapper with the old name is provided for compatibility.

_parse_m3u8_formats is likewise renamed and extended, but without adding
the compatibility wrapper; the test suite is adjusted to test the enhanced
method instead.
2021-04-28 17:19:57 +05:30
Felix S
19bb39202d [extractor/common] Generalise _merge_subtitles
This allows modifying a subtitles dictionary in-place.
2021-04-28 17:19:46 +05:30
pukkandan
6a04a74e8b
[FormatSort] Fix for when some formats have quality and others don't 2021-04-26 10:56:54 +05:30
pukkandan
b7da73eb19
Add option --ignore-no-formats-error
* Ignores the "no video format" and similar errors
* Experimental - Some extractors may still throw these errors
2021-04-17 08:40:30 +05:30