ls_dlp
ls-dlp
This is a forked project of livestream_dl.
A robust YouTube livestream downloader that combines the reliability of yt-dlp with the fragment-based downloading principles of ytarchive.
This tool focuses on using yt-dlp for stream information extraction, ensuring compatibility with YouTube's changing infrastructure. It is designed to handle live recording with the ability to recover streams if they become unavailable (e.g., go private) during the download process.
Key Features
- Robust Downloading: Uses
yt-dlpfor extraction, minimizing breakage from YouTube updates. - Stream Recovery: capable of recovering streams that go private during recording (requires specific configuration).
- Dual Write Modes: Supports both direct
.tsfile writing (low disk IO) and SQLite-backed downloading (safer for unstable connections). - Live Chat: Integrated support for downloading live chat (via
yt-dlp) and bundling it into a ZIP file. - Channel Monitoring: Automated monitoring of channels for upcoming or live streams.
- Protocol Fallbacks: Optional support for DASH and HLS (m3u8) protocols to avoid modifying
yt-dlpsource code.
Requirements
See Dockerfile
Installation
pip install ls-dlp
Usage
ls-dlp [OPTIONS] [VIDEO_URL_OR_ID]
Examples
Basic download:
ls-dlp \
--threads 4 \
--dash \
--m3u8 \
--write-thumbnail \
--embed-thumbnail \
--wait-for-video "60:600" \
--clean-info-json \
--remove-ip-from-json \
--live-chat \
--resolution "best" \
--write-info-json \
--log-level "INFO" -- "[VIDEO_ID]"
Download with live chat and embed thumbnail:
ls-dlp --live-chat --threads 4 --embed-thumbnail VIDEO_ID
Monitor a channel for streams:
ls-dlp --monitor-channel --threads 4 --dash --m3u8 --wait-for-video 60 CHANNEL_ID
Command Line Options
General Configuration
| Option | Default | Description |
|---|---|---|
ID |
None |
The video URL or ID (Positional argument). |
--help, -h |
N/A | Show the help message and exit. |
--cookies |
None |
Path to a Netscape-formatted cookies file (required for age-gated or members-only content). |
--temp-folder |
None |
Path for temporary files (database, segments). Supports yt-dlp output formatting. |
--output |
%(fulltitle)s (%(id)s) |
Path/filename for output files. Supports yt-dlp output formatting. A good example is: "[%(release_date,upload_date)s] %(fulltitle).120B [%(channel)s] (%(id)s)" |
--ext |
None |
Force the extension of the final video file (e.g., .mp4, .mkv). |
--json-file |
None |
Path to an existing yt-dlp info.json file. Overrides ID and skips retrieving URLs from the web. |
--write-ffmpeg-command |
False |
Writes the final FFmpeg command used to merge the file to a .txt file for debugging or manual merging. |
Download & Formatting
| Option | Default | Description |
|---|---|---|
--resolution |
best |
Desired resolution. Can be best, audio_only, or a custom yt-dlp format filter. -Audio is always set to "ba" (best audio) regardless of filters. -"best" is converted to "bv". -If unspecified, a prompt will appear. |
--custom-sort |
None |
Custom sorting algorithm for formats based on yt-dlp sorting syntax. |
--threads |
1 |
Number of download threads per format. Total threads = this value × 2 (for video+audio). |
--batch-size |
5 |
Number of segments downloaded before the temporary database is committed to disk (reduces disk IO). |
--segment-retries |
10 |
Number of times to retry downloading a specific segment before failing. |
--dash |
False |
Use DASH URLs as fallback. Does not require yt-dlp modification but prevents stream recovery if DASH URLs are used. |
--m3u8 |
False |
Use HLS (m3u8) URLs as fallback. Combined audio/video streams (halves requests). Does not require yt-dlp modification but prevents recovery if m3u8 URLs are used. |
--force-m3u8 |
False |
Forces the use of m3u8 stream URLs. |
--wait-for-video |
None |
Wait time (int) or range (min:max) to wait for a video to start or become available. Refer to yt-dlp's documentation |
--ytdlp-options |
None |
JSON string of additional yt-dlp options (overwrites conflicts). E.g '{"extractor_args":{"youtubepot-bgutilhttp":{"base_url":["http://bgutil-provider:4416"]}}}' to use Potoken retrieval. Refer to full options for more examples. |
Processing & Output
| Option | Default | Description |
|---|---|---|
--no-merge |
False |
Do not merge video and audio using FFmpeg after download. |
--merge |
False |
Force merging video using FFmpeg (overrides --no-merge). |
--write-thumbnail |
False |
Save the video thumbnail to a separate file. |
--embed-thumbnail |
False |
Embed the thumbnail into the final video file (ignored if --no-merge is active). |
--write-info-json |
False |
Write the video metadata to an info.json file. |
--write-description |
False |
Write the video description to a separate text file. |
--keep-temp-files |
False |
Keep all temporary files (database and TS files) after finishing. |
--keep-ts-files |
False |
Keep the intermediate TS files but delete the database. |
--live-chat |
False |
Download live chat (requires yt-dlp). |
--stop-chat-when-done |
300 |
Max seconds to wait for chat download to finish after stream ends. |
Database & Storage Modes
| Option | Default | Description |
|---|---|---|
--database-in-memory |
False |
Keep the segment database in RAM. High memory usage; not recommended for long streams. |
--direct-to-ts |
False |
Write directly to a .ts file instead of using a SQLite database. See "Downloader methods" below. |
--keep-database-file |
False |
Keep the database file after completion. If using --direct-to-ts, keeps the state file. |
Recovery & Resilience
| Option | Default | Description |
|---|---|---|
--recovery |
False |
Puts the downloader directly into stream recovery mode. |
--force-recover-merge |
False |
Forces merging to the final file even if not all segments were successfully recovered. |
--recovery-failure-tolerance |
0 |
Max number of fragments allowed to fail during recovery without throwing a final error. |
--wait-limit |
0 |
Max wait intervals (~10s each) for new segments. If 0, waits until status changes to post_live or was_live. |
Network & Logging
| Option | Default | Description |
|---|---|---|
--proxy |
None |
Proxy URL (string) or JSON string for multiple methods. First proxy used for yt-dlp/chat. |
--ipv4 |
False |
Force IPv4 only. |
--ipv6 |
False |
Force IPv6 only. |
--log-level |
INFO |
Logging level: DEBUG, VERBOSE, INFO, WARNING, ERROR, CRITICAL. Verbose logging is a custom level that includes the INFO logs of yt-dlp. |
--no-console |
False |
Disable printing log messages to the console. |
--log-file |
None |
Path to a file where log messages will be saved. |
--stats-as-json |
False |
Prints download statistics as a JSON string (bypasses log level). |
--new-line |
False |
Ensures console messages always print to a new line (useful for some terminals). |
Channel Monitoring
| Option | Default | Description |
|---|---|---|
--monitor-channel |
False |
Enables channel monitoring mode. Use the Channel ID in the ID argument. |
--members-only |
False |
Monitor the 'Members Only' playlist instead of public streams. Requires cookies. |
--upcoming-lookahead |
24 |
Maximum time (in hours) to look ahead for upcoming videos to schedule. |
--playlist-items |
50 |
Maximum number of playlist items to check when monitoring. |
Metadata Privacy
| Option | Default | Description |
|---|---|---|
--remove-ip-from-json |
False |
Replaces IP addresses in info.json with 0.0.0.0. |
--clean-urls |
False |
Removes potentially identifiable stream URLs from info.json. |
--clean-info-json |
False |
Enables yt-dlp's internal clean-info-json option. |
Full options
$ ls-dlp --help
usage: runner.py [-h] [--resolution RESOLUTION] [--custom-sort CUSTOM_SORT] [--threads THREADS] [--batch-size BATCH_SIZE] [--segment-retries SEGMENT_RETRIES] [--no-merge] [--merge] [--cookies COOKIES] [--output OUTPUT] [--ext EXT] [--temp-folder TEMP_FOLDER] [--write-thumbnail] [--embed-thumbnail]
[--write-info-json] [--write-description] [--keep-temp-files] [--keep-ts-files] [--live-chat] [--keep-database-file] [--recovery] [--force-recover-merge] [--recovery-failure-tolerance RECOVERY_FAILURE_TOLERANCE] [--wait-limit WAIT_LIMIT] [--database-in-memory] [--direct-to-ts]
[--wait-for-video WAIT_FOR_VIDEO] [--json-file JSON_FILE] [--remove-ip-from-json] [--clean-urls] [--clean-info-json] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-console] [--log-file LOG_FILE] [--write-ffmpeg-command] [--stats-as-json] [--ytdlp-options YTDLP_OPTIONS] [--dash]
[--m3u8] [--force-m3u8] [--proxy [PROXY]] [--ipv4 | --ipv6] [--stop-chat-when-done STOP_CHAT_WHEN_DONE] [--new-line] [--monitor-channel] [--members-only] [--upcoming-lookahead UPCOMING_LOOKAHEAD] [--playlist-items PLAYLIST_ITEMS]
[ID]
Download YouTube livestream (https://github.com/CanOfSocks/livestream_dl)
positional arguments:
ID The video URL or ID
options:
-h, --help show this help message and exit
--resolution RESOLUTION
Desired resolution. Can be best, audio_only or a custom filter based off yt-dlp's format filtering: https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#filtering-formats. Audio will always be set as "ba" (best audio) regardless of filters set. "best" will be converted to "bv" A prompt
will be displayed if no value is entered
--custom-sort CUSTOM_SORT
Custom sorting algorithm for formats based off yt-dlp's format sorting: https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#sorting-formats
--threads THREADS Number of download threads per format. This will be 2x for an video and audio download. Default: 1
--batch-size BATCH_SIZE
Number of segments before the temporary database is committed to disk. This is useful for reducing disk access instances. Default: 5
--segment-retries SEGMENT_RETRIES
Number of times to retry grabbing a segment. Default: 10
--no-merge Don't merge video using ffmpeg
--merge Merge video using ffmpeg, overrides --no-merge
--cookies COOKIES Path to cookies file
--output OUTPUT Path/file name for output files. Supports yt-dlp output formatting
--ext EXT Force extension of video file. E.g. '.mp4'
--temp-folder TEMP_FOLDER
Path for temporary files. Supports yt-dlp output formatting
--write-thumbnail Write thumbnail to file
--embed-thumbnail Embed thumbnail into final file. Ignored if --no-merge is used
--write-info-json Write info.json to file
--write-description Write description to file
--keep-temp-files Keep all temp files i.e. database and/or ts files
--keep-ts-files Keep all ts files
--live-chat Get Live chat
--keep-database-file Keep database file. If using with --direct-to-ts, this keeps the state file
--recovery Puts downloader into stream recovery mode
--force-recover-merge
Forces merging to final file even if all segments could not be recovered
--recovery-failure-tolerance RECOVERY_FAILURE_TOLERANCE
Maximum number of fragments that fail to download (exceed the retry limit) and not throw an error. May cause unexpected issues when merging to .ts file and remuxing. Default: 0
--wait-limit WAIT_LIMIT
Set maximum number of wait intervals for new segments. Each wait interval is ~10s (e.g. a value of 20 would be 200s). A minimum of value of 20 is recommended. Stream URLs are refreshed every 10 intervals. A value of 0 wait until the video moves into 'was_live' or 'post_live' status.
Default: 0
--database-in-memory Keep stream segments database in memory. Requires a lot of RAM (Not recommended)
--direct-to-ts Write directly to ts file instead of database. May use more RAM if a segment is slow to download. This overwrites most database options
--wait-for-video WAIT_FOR_VIDEO
Wait time (int) or Minimum and maximum (min:max) interval to wait for a video
--json-file JSON_FILE
Path to existing yt-dlp info.json file. Overrides ID and skips retrieving URLs
--remove-ip-from-json
Replaces IP entries in info.json with 0.0.0.0
--clean-urls Removes stream URLs from info.json that contain potentially identifiable information. These URLs are usually useless once they have expired
--clean-info-json Enables yt-dlp's 'clean-info-json' option
--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
Set the logging level. Default is INFO.
--no-console Do not log messages to the console.
--log-file LOG_FILE Path to the log file where messages will be saved.
--write-ffmpeg-command
Writes FFmpeg command to a txt file
--stats-as-json Prints stats as a JSON formatted string. Bypasses logging and prints regardless of log level
--ytdlp-options YTDLP_OPTIONS
Additional yt-dlp options as a JSON string. Overwrites any options that are already defined by other options. Available options: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py#L183. E.g. '{"extractor_args": {"youtube": {"player_client": ["web_creator"]}, "youtubepot-
bgutilhttp":{ "base_url": ["http://10.1.1.40:4416"]}}}' if you have installed the potoken plugin
--dash Gets any available DASH urls as a fallback to adaptive URLs. Dash URLs do not require yt-dlp modification to be used, but can't be used for stream recovery and can cause large info.json files when a stream is in the 'post_live' status
--m3u8 Gets any available m3u8 urls as a fallback to adaptive URLs. m3u8 URLs do not require yt-dlp modification to be used, but can't be used for stream recovery. m3u8 URLs provide both video and audio in each fragment and could allow for the amount of segment download requests to be halved
--force-m3u8 Forces use of m3u8 stream URLs
--proxy [PROXY] (Requires testing) Specify proxy to use for web requests. Can be a string for a single proxy or a JSON formatted string to specify multiple methods. For multiple, refer to format https://requests.readthedocs.io/en/latest/user/advanced/#proxies. The first proxy specified will be used for
yt-dlp and live chat functions.
--ipv4 Force IPv4 only
--ipv6 Force IPv6 only
--stop-chat-when-done STOP_CHAT_WHEN_DONE
Wait a maximum of X seconds after a stream is finished to download live chat. Default: 300. This is useful if waiting for chat to end causes hanging.
--new-line Console messages always print to new line. (Currently only ensured for stats output)
--monitor-channel Use monitor channel feature (Alpha). Specify channel ID in 'ID' argument
--members-only Monitor 'Members Only' playlist for streams instead of 'Streams' playlist. Requires cookies.
--upcoming-lookahead UPCOMING_LOOKAHEAD
Maximum time (in hours) to start a downloader instance for a video. Default: 24
--playlist-items PLAYLIST_ITEMS
Maximum number of playlist items to check. Default: 50
Downloader methods
There are two download methods available for livestream downloading, using a SQLite database and writing directly to a ts file. For stream recovery, only the SQLite option is available. Merging to a video or audio file (.mp4, .mkv, .ogg etc) will require an extra write of information. The default method is the SQLite method.
Direct writing to a .ts file
Writing directly to a ts file helps reduce the number of writes to a disk during the recording as it is only necessary to write to the final ts file. This may reduce disk wear over a long period of time of downloading many streams. As segments are downloaded, the sequence number of the segment is used to decide which segment will be appended to the ts file next. If the "next" segment is not available to be written to the disk, other segments will be stored in RAM until the respective previous segment has been written to the disk. This performs in a similar way to ytarchive. To reduce file opening and closing actions on the file system, if multiple sequential segments can be written at once, they will be written into the file at once. A state file is saved each time segments are written to the disk, saving the latest segment and size of the file, so the downloader doesn't need to re-download segments should the downloading session stop for any reason. Segment downloads are not guaranteed to finish in a sequential order if more than 1 thread is used. If a segment is particularly slow at downloading, this may increase ram usage significantly while subsequent segments are saved in RAM.
For regular livestream recording that doesn't experience frequent segment download errors, using the direct writing method will work well.
SQLite
The SQLite downloader method is used to improve handling of non-sequential successful segment downloads. This works by creating and using a basic table of the segment number as an ID and a blob to store the segment data. Once all segments are downloaded, a query is executed to sort all of the downloaded segments into the correct order and saved to a .ts file. This helps significantly to manage downloaded segments when failures occur often, such as unavailable stream recovery. By writing to the database and a final .ts file, this will require 2 full writes of the downloaded data, which may increase wear on flash-based systems. For most users, this increased wear will not be significant in the long-term. This improves over saving individual segment files like ytarchive-raw-go as all the downloaded segments are encapsulated into a single file, making it easier for the file system to handle. In the SQLite method, the existence of a downloaded segment is checked before it is downloaded. This allows failed segments to be "looped back to" later without causing other slowdowns, and ensures some information is saved for a segment (even if it is empty, as is the case sometimes).
Known Issues
- Concurrent Futures: The downloader uses
concurrent.futuresfor thread management. While robust, stopping the downloader gracefully (e.g., via Keyboard Interrupt) can sometimes be delayed if a thread is stuck on a request. - Stream Recovery: This feature is currently in a "semi-broken" state and may not work reliably for all stream types.
1""".. include:: ../README.md""" 2 3from __future__ import annotations 4 5import importlib.metadata 6 7from .download_live import LiveStreamDownloader 8 9try: 10 __version__ = importlib.metadata.version(__name__) 11except importlib.metadata.PackageNotFoundError: 12 __version__ = "0.0.0" 13 14 15__all__ = ( 16 "LiveStreamDownloader", 17 "__version__", 18)
39class LiveStreamDownloader: 40 def __init__( 41 self, 42 kill_all: threading.Event = threading.Event(), 43 cleanup: threading.Event = threading.Event(), 44 logger: logging.Logger | logging.LoggerAdapter | None = None, 45 kill_this: threading.Event | None = None, 46 ) -> None: 47 if logger: 48 self.logger = logger 49 else: 50 # 1. Create a Named Logger instance and assign it to self.logger 51 self.logger = logging.getLogger(self.__class__.__name__) 52 # Ensure it processes messages (important if you don't call setup_logging immediately) 53 if isinstance(self.logger, logging.Logger): 54 self.logger.propagate = False 55 self.kill_this = kill_this or threading.Event() 56 57 # Global state converted to instance attributes 58 self.kill_all = kill_all 59 self.cleanup = cleanup 60 self.live_chat_result = None 61 self.chat_timeout = None 62 # File name dictionary 63 self.file_names = {"databases": [], "streams": {}} 64 self.stats = {} 65 66 self.refresh_json = {} 67 self.live_status = "" 68 self.lock: threading.Lock = threading.Lock() 69 70 # Create runner function for each download format 71 def download_stream( 72 self, 73 info_dict, 74 resolution, 75 batch_size=5, 76 max_workers=1, 77 folder=None, 78 file_name=None, 79 keep_database=False, 80 cookies=None, 81 retries=5, 82 yt_dlp_options=None, 83 proxies=None, 84 yt_dlp_sort=None, 85 include_dash=False, 86 include_m3u8=False, 87 force_m3u8=False, 88 manifest=0, 89 **kwargs, 90 ): 91 try: 92 download_params = locals().copy() 93 download_params.update({"download_function": self.download_stream}) 94 file = None 95 filetype = None 96 97 with DownloadStream( 98 info_dict, 99 resolution=resolution, 100 batch_size=batch_size, 101 max_workers=max_workers, 102 folder=folder, 103 file_name=file_name, 104 cookies=cookies, 105 fragment_retries=retries, 106 yt_dlp_options=yt_dlp_options, 107 proxies=proxies, 108 yt_dlp_sort=yt_dlp_sort, 109 include_dash=include_dash, 110 include_m3u8=include_m3u8, 111 force_m3u8=force_m3u8, 112 download_params=download_params, 113 livestream_coordinator=self, 114 **kwargs, 115 ) as downloader: 116 self.stats["status"] = "Recording" 117 downloader.live_dl() 118 file_name = downloader.combine_segments_to_file(downloader.merged_file_name) 119 if not keep_database: 120 self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}") 121 downloader.delete_temp_database() 122 elif downloader.temp_db_file != ":memory:": 123 database_file = FileInfo(downloader.temp_db_file, file_type="database", format=downloader.format) 124 self.file_names["databases"].append(database_file) 125 126 file = FileInfo(file_name, file_type=downloader.type, format=downloader.format) 127 filetype = downloader.type 128 129 self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file}) 130 131 return file, filetype 132 except Exception: 133 self.logger.exception("Unexpected error occurred while downloading stream") 134 raise 135 136 # Create runner function for each download format 137 def download_stream_direct( 138 self, 139 info_dict, 140 resolution, 141 batch_size, 142 max_workers, 143 folder=None, 144 file_name=None, 145 keep_state=False, 146 cookies=None, 147 retries=5, 148 yt_dlp_options=None, 149 proxies=None, 150 yt_dlp_sort=None, 151 include_dash=False, 152 include_m3u8=False, 153 force_m3u8=False, 154 manifest=0, 155 **kwargs, 156 ): 157 try: 158 download_params = locals().copy() 159 download_params.update({"download_function": self.download_stream_direct}) 160 file = None 161 filetype = None 162 163 with DownloadStreamDirect( 164 info_dict, 165 resolution=resolution, 166 max_workers=max_workers, 167 folder=folder, 168 file_name=file_name, 169 cookies=cookies, 170 fragment_retries=retries, 171 yt_dlp_options=yt_dlp_options, 172 proxies=proxies, 173 yt_dlp_sort=yt_dlp_sort, 174 include_dash=include_dash, 175 include_m3u8=include_m3u8, 176 force_m3u8=force_m3u8, 177 download_params=download_params, 178 livestream_coordinator=self, 179 **kwargs, 180 ) as downloader: 181 self.stats["status"] = "Recording" 182 file_name = downloader.live_dl() 183 file = FileInfo(file_name, file_type=downloader.type, format=downloader.format) 184 filetype = downloader.type 185 downloader.delete_state_file() 186 187 self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file}) 188 return file, filetype 189 except Exception: 190 self.logger.exception("Unexpected error occurred while downloading stream") 191 raise 192 193 def recover_stream( 194 self, 195 info_dict, 196 resolution, 197 batch_size=5, 198 max_workers=5, 199 folder=None, 200 file_name=None, 201 keep_database=False, 202 cookies=None, 203 retries=5, 204 yt_dlp_options=None, 205 proxies=None, 206 yt_dlp_sort=None, 207 force_merge=False, 208 recovery_failure_tolerance=0, 209 manifest=0, 210 stream_urls: list | None = None, 211 no_merge=False, 212 **kwargs, 213 ): 214 if stream_urls is None: 215 stream_urls = [] 216 try: 217 file = None 218 filetype = None 219 220 with StreamRecovery( 221 info_dict, 222 resolution=resolution, 223 batch_size=batch_size, 224 max_workers=max_workers, 225 folder=folder, 226 file_name=file_name, 227 cookies=cookies, 228 fragment_retries=retries, 229 proxies=proxies, 230 yt_dlp_sort=yt_dlp_sort, 231 livestream_coordinator=self, 232 stream_urls=stream_urls, 233 **kwargs, 234 ) as downloader: 235 self.stats["status"] = "Recording" 236 result = downloader.live_dl() 237 if (force_merge or result <= 0 or result <= recovery_failure_tolerance) and (not no_merge): 238 if result > 0: 239 self.logger.warning( 240 f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting" 241 ) 242 file_name = downloader.combine_segments_to_file(downloader.merged_file_name) 243 if not keep_database: 244 self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}") 245 downloader.delete_temp_database() 246 elif downloader.temp_db_file != ":memory:": 247 database_file = FileInfo( 248 downloader.temp_db_file, file_type="database", format=downloader.format 249 ) 250 self.file_names["databases"].append(database_file) 251 else: 252 msg = f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting" 253 raise VideoDownloadError(msg) 254 file = FileInfo(file_name, file_type=downloader.type, format=downloader.format) 255 filetype = downloader.type 256 257 self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file}) 258 except Exception: 259 self.logger.exception("Unexpected error occurred while downloading stream") 260 raise 261 return file, filetype 262 263 def submit_download( 264 self, executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False 265 ): 266 extra_kwargs = {} 267 268 # Options only not used by recovery 269 if not options.get("recovery", False): 270 extra_kwargs.update( 271 { 272 "include_dash": options.get("dash", False), 273 "include_m3u8": options.get("m3u8", False), 274 "force_m3u8": options.get("force_m3u8", False), 275 } 276 ) 277 278 # Select the function 279 if options.get("recovery", False): 280 func = self.recover_stream 281 extra_kwargs.update( 282 { 283 "force_merge": options.get("force_recovery_merge", False), 284 "recovery_failure_tolerance": options.get("recovery_failure_tolerance", 0), 285 } 286 ) 287 elif options.get("direct_to_ts", False): 288 func = self.download_stream_direct 289 extra_kwargs.update( 290 { 291 "keep_state": options.get("keep_temp_files", False) or options.get("keep_database_file", False), 292 } 293 ) 294 else: 295 func = self.download_stream 296 extra_kwargs.update( 297 { 298 "keep_database": options.get("keep_temp_files", False) or options.get("keep_database_file", False), 299 } 300 ) 301 302 kwargs = { 303 "info_dict": info_dict, 304 "resolution": "audio_only" if is_audio else resolution, 305 "batch_size": options.get("batch_size", 1), 306 "max_workers": options.get("threads", 1), 307 "folder": download_folder, 308 "file_name": file_name, 309 "retries": options.get("segment_retries"), 310 "cookies": options.get("cookies"), 311 "yt_dlp_options": options.get("ytdlp_options", None), 312 "proxies": options.get("proxy", None), 313 "yt_dlp_sort": options.get("custom_sort", None), 314 } 315 316 if extra_kwargs: 317 kwargs.update(extra_kwargs) 318 319 # Add any extra values not yet existing 320 for key, value in options.items(): 321 kwargs.setdefault(key, value) 322 323 self.logger.debug(f"Starting executor with: {json.dumps(kwargs)}") 324 325 # Submit to executor 326 future = executor.submit(func, **kwargs) 327 futures.add(future) 328 return future 329 330 # Multithreaded function to download new segments with delayed commit after a batch 331 def download_segments( 332 self, info_dict, resolution="best", options=None, thread_event: threading.Event | None = None 333 ) -> None: 334 if options is None: 335 options = {} 336 futures = set() 337 338 if thread_event is not None: 339 self.kill_all = thread_event 340 341 self.stats["id"] = info_dict.get("id", None) 342 self.stats["status"] = "Starting" 343 self.logger.debug(json.dumps(options, indent=4)) 344 outputFile = self.output_filename(info_dict=info_dict, outtmpl=options.get("output")) 345 file_name = None 346 # Requires testing 347 if options.get("temp_folder") is not None and options.get("temp_folder") != os.path.dirname(outputFile): 348 _output_folder, file_name = os.path.split(outputFile) 349 download_folder = self.output_filename(info_dict=info_dict, outtmpl=options.get("temp_folder")) 350 options["temp_folder"] = download_folder 351 else: 352 download_folder, file_name = os.path.split(outputFile) 353 options["filename"] = file_name 354 if download_folder: 355 os.makedirs(download_folder, exist_ok=True) 356 357 with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor: 358 done = not_done = set() 359 try: 360 # Download auxiliary files (thumbnail, info,json etc) 361 auxiliary_thread = executor.submit(self.download_auxiliary_files, info_dict=info_dict, options=options) 362 futures.add(auxiliary_thread) 363 364 live_chat_thread = None 365 if options.get("live_chat", False) is True: 366 live_chat_thread = threading.Thread( 367 target=self.download_live_chat, args=(info_dict, options), daemon=True 368 ) 369 live_chat_thread.start() 370 371 format_check = Formats().getFormatURL( 372 info_json=info_dict, 373 resolution=resolution, 374 sort=options.get("custom_sort"), 375 include_dash=(options.get("dash", False) and not options.get("recovery", False)), 376 include_m3u8=options.get("m3u8", False), 377 force_m3u8=options.get("force_m3u8", False), 378 ) 379 if not format_check: 380 msg = "Resolution is not valid or does not exist in stream" 381 raise ValueError(msg) 382 # For use of specificed format. Expects two values, but can work with more 383 # Video + Audio 384 385 if resolution.lower() not in ("audio_only", "ba"): 386 # Video 387 self.submit_download( 388 executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False 389 ) 390 if format_check.protocol != "m3u8_native": 391 # Audio 392 self.submit_download( 393 executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True 394 ) 395 # Audio-only 396 else: 397 self.submit_download( 398 executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True 399 ) 400 401 while True: 402 if self.kill_all.is_set() or self.kill_this.is_set(): 403 msg = "Thread kill event is set, ending..." 404 raise KeyboardInterrupt(msg) 405 done, not_done = concurrent.futures.wait( 406 futures, timeout=1, return_when=concurrent.futures.FIRST_COMPLETED 407 ) 408 # Continuously check for completion or interruption 409 for future in done: 410 result, type = future.result() 411 self.logger.info(f"\033[31m{result}\033[0m") 412 413 if type == "auxiliary": 414 self.file_names.update(result) 415 416 # Remove parts that would not be needed by stream downloaders 417 self.remove_format_segment_playlist_from_info_dict(info_dict=info_dict) 418 info_dict.pop("thumbnails", None) 419 # Remove no longer required parts for the rest of the downloader 420 # info_dict = self.trim_info_json(info_dict=info_dict, keys_to_keep={"id", "ext", "upload_date", "original_url", "description", "fulltitle", "channel"}) 421 422 elif str(type).lower() == "video": 423 # self.file_names['video'] = result 424 pass 425 elif str(type).lower() == "audio": 426 # self.file_names['audio'] = result 427 pass 428 elif str(type).lower() == "live_chat": 429 pass 430 else: 431 self.file_names[str(type)] = result 432 433 futures.discard(future) 434 435 if len(not_done) <= 0: 436 break 437 # else: 438 # time.sleep(0.9) 439 self.print_stats(options=options) 440 441 if live_chat_thread and live_chat_thread.is_alive(): 442 self.logger.info("Media downloads finished, waiting for live chat to end...") 443 if options.get("stop_chat_when_done"): 444 self.chat_timeout = time.time() 445 while live_chat_thread.is_alive() and not (self.kill_all.is_set() or self.kill_this.is_set()): 446 time.sleep(0.1) 447 448 except KeyboardInterrupt: 449 self.kill_this.set() 450 self.logger.debug("Keyboard interrupt detected") 451 if len(not_done) > 0: 452 self.logger.debug("Cancelling remaining threads") 453 for future in not_done: 454 _ = future.cancel() 455 done, not_done = concurrent.futures.wait( 456 futures, timeout=5, return_when=concurrent.futures.ALL_COMPLETED 457 ) 458 executor.shutdown(wait=False, cancel_futures=True) 459 self.logger.debug("Shutdown all threads") 460 self.stats["status"] = "Cancelled" 461 raise 462 463 self.stats["status"] = "Muxing" 464 self.create_mp4(file_names=self.file_names, info_dict=info_dict, options=options) 465 466 self.stats["status"] = "Moving" 467 self.move_to_final(options=options, output_file=outputFile, file_names=self.file_names) 468 self.stats["status"] = "Finished" 469 # move_to_final(info_dict, options, file_names) 470 471 def output_filename(self, info_dict, outtmpl): 472 return str(YoutubeDL().prepare_filename(info_dict, outtmpl=outtmpl)).replace( 473 "%", "%" 474 ) # Replace normal percent sign with unicode version to prevent ffmpeg errors 475 476 def move_to_final(self, options, output_file, file_names) -> None: 477 def maybe_move(key, dest_func, file_names_dict=file_names, option_flag=None) -> None: 478 """key: key in file_names 479 dest_func: func -> pathlib -> string dest path 480 delete_if_false: if True, delete when option_flag not set 481 option_flag: name of boolean option (write_thumbnail etc) 482 """ 483 f = file_names_dict.get(key) 484 if not f: 485 return 486 try: 487 # deletion case (thumbnail / ffmpeg) 488 if option_flag is not None and not options.get(option_flag, False): 489 self.logger.info(f"Removing {f.absolute()}") 490 f.unlink(missing_ok=True) 491 file_names_dict.pop(key, None) 492 return 493 494 dest = dest_func(f) 495 if str(f).strip() != str(dest).strip(): 496 self.logger.info(f"Moving {f.absolute()} → {dest}") 497 shutil.move(f.absolute(), dest) 498 else: 499 self.logger.debug(f"{f.absolute()} is already in final destination") 500 except Exception as e: 501 self.logger.exception(f"unable to move {key}: {e}") 502 503 # ensure output dir exists 504 out_dir = os.path.dirname(output_file) 505 if out_dir: 506 os.makedirs(out_dir, exist_ok=True) 507 508 # === individual file handlers === 509 510 maybe_move("thumbnail", lambda f: f"{output_file}{f.suffix}", option_flag="write_thumbnail") 511 512 maybe_move( 513 "info_json", 514 lambda f: f"{output_file}.info.json", 515 ) 516 517 maybe_move( 518 "description", 519 lambda f: f"{output_file}{f.suffix}", 520 ) 521 522 stream_manifests = list(self.file_names["streams"].items()) 523 for manifest, stream in stream_manifests: 524 stream_output_file = output_file 525 if len(stream_manifests) > 1: 526 stream_output_file = f"{output_file}.{manifest}" 527 maybe_move("video", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream) 528 529 maybe_move("audio", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream) 530 531 maybe_move("merged", lambda f: f"{stream_output_file}{f.suffix}", file_names_dict=stream) 532 533 maybe_move( 534 "live_chat", 535 lambda f: f"{output_file}.live_chat.zip", 536 ) 537 538 # special: databases = list 539 try: 540 for f in file_names.get("databases", []): 541 dest = f"{output_file}.{f._format}{f.suffix}" 542 if str(f).strip() != str(dest).strip(): 543 self.logger.info(f"Moving {f.absolute()} → {dest}") 544 shutil.move(f.absolute(), dest) 545 except Exception as e: 546 self.logger.exception(f"unable to move database files: {e}") 547 548 # special: ffmpeg_cmd 549 maybe_move("ffmpeg_cmd", lambda f: f"{output_file}.ffmpeg.txt", option_flag="write_ffmpeg_command") 550 551 # remove temp folder 552 """ 553 if options.get('temp_folder', None) is not None: 554 try: 555 os.rmdir(options.get('temp_folder')) 556 except OSError as e: 557 if e.errno == errno.ENOTEMPTY: 558 self.logger.warning(f"Error: Directory not empty: {e.filename}") 559 else: 560 self.logger.exception(f"Error removing temp folder: {e}") 561 except Exception as e: 562 self.logger.exception(f"Error removing temp folder: {e}") 563 """ 564 self.logger.info("Finished moving files from temporary directory to output destination") 565 566 def download_live_chat(self, info_dict, options): 567 filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id") 568 569 base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename 570 571 class YTDLP_Chat_logger(YTDLPLogger): 572 def __init__(self, logger: logging.Logger | logging.LoggerAdapter | None = None) -> None: 573 if logger is None: 574 logger = logging.getLogger() 575 super().__init__(logger=logger) 576 577 def prefix(self, msg: str): 578 if msg.startswith("[live-chat] "): 579 return msg 580 return f"[live-chat] {msg}" 581 582 def debug(self, msg) -> None: 583 super().debug(self.prefix(msg)) 584 585 def info(self, msg) -> None: 586 super().info(self.prefix(msg)) 587 588 def warning(self, msg) -> None: 589 super().warning(self.prefix(msg)) 590 591 def error(self, msg) -> None: 592 super().error(self.prefix(msg)) 593 594 logger = self.logger 595 596 ydl_opts = { 597 "skip_download": True, # Skip downloading video/audio 598 "logger": YTDLP_Chat_logger(logger=logger), 599 #'quiet': True, 600 "cookiefile": options.get("cookies", None), 601 "retries": 25, 602 "concurrent_fragment_downloads": 3, 603 #'live_from_start': True, 604 "writesubtitles": True, # Extract subtitles (live chat) 605 "subtitlesformat": "json", # Set format to JSON 606 "subtitleslangs": ["live_chat"], # Only extract live chat subtitles 607 "outtmpl": base_output, # Save to a JSON file 608 } 609 self.logger.debug(options.get("ytdlp_options", {})) 610 ydl_opts.update(options.get("ytdlp_options", {})) 611 612 livechat_filename = base_output + ".live_chat.json" 613 614 self.logger.info(f"Downloading live chat to: {livechat_filename}") 615 616 # Download live chat using yt-dlp 617 with YoutubeDL(ydl_opts) as ydl: 618 try: 619 ydl.download([info_dict.get("original_url", info_dict.get("webpage_url", info_dict.get("url")))]) 620 except Exception as e: 621 self.logger.exception(f"Failed to download live chat: {e}") 622 623 time.sleep(1) 624 part_file = f"{livechat_filename}.part" 625 if os.path.exists(part_file): 626 # Append part to 627 chunk_size = 1024 * 1024 * 10 # number of characters per chunk, up to 10M characters 628 with open(part_file, encoding="utf-8") as fa, open(livechat_filename, "a", encoding="utf-8") as fb: 629 while chunk := fa.read(chunk_size): 630 fb.write(chunk) 631 os.remove(f"{livechat_filename}.part") 632 633 try: 634 if not os.path.exists(livechat_filename): 635 self.logger.warning(f"Live chat file not found: {livechat_filename}") 636 return None, None 637 638 zip_filename = base_output + ".live_chat.zip" 639 with zipfile.ZipFile( 640 zip_filename, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=9, allowZip64=True 641 ) as zipf: 642 zipf.write(livechat_filename, arcname=os.path.basename(livechat_filename)) 643 os.remove(livechat_filename) 644 live_chat = {"live_chat": FileInfo(zip_filename, file_type="live_chat")} 645 self.file_names.update(live_chat) 646 return live_chat, "live_chat" 647 except Exception as e: 648 self.logger.exception(f"\033[31m{e}\033[0m") 649 650 def replace_ip_in_json(self, file_name) -> None: 651 pattern = re.compile(r"((?:[0-9]{1,3}\.){3}[0-9]{1,3})|((?:[a-f0-9]{1,4}:){7}[a-f0-9]{1,4})") 652 653 with open(file_name, encoding="utf-8") as file: 654 content = file.read() 655 656 modified_content = re.sub(pattern, "0.0.0.0", content) 657 658 with open(file_name, "w", encoding="utf-8") as file: 659 file.write(modified_content) 660 661 def remove_urls_from_json(self, file_name) -> None: 662 with open(file_name, encoding="utf-8") as file: 663 data = json.load(file) 664 665 if data.get("formats", None) is not None: 666 for format in data["formats"]: 667 if format.get("url") is not None: 668 format["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 669 670 if format.get("manifest_url") is not None: 671 format["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 672 673 format.pop("fragment_base_url", None) 674 format.pop("fragments", None) 675 676 if data.get("thumbnails", None) is not None: 677 for thumbnail in data["thumbnails"]: 678 if thumbnail.get("url", None) is not None: 679 parsed_url = urlparse(thumbnail.get("url", "")) 680 thumbnail["url"] = f"{parsed_url.scheme}://{parsed_url.netloc}{parsed_url.path}" 681 682 if data.get("url", None) is not None: 683 data["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 684 685 if data.get("manifest_url", None) is not None: 686 data["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 687 688 data["removed_urls"] = True 689 690 with open(file_name, "w", encoding="utf-8") as file: 691 json.dump(data, file) 692 693 def download_auxiliary_files(self, info_dict, options): 694 filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id") 695 696 base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename 697 698 created_files = {} 699 700 logger = self.logger 701 702 class YTDLP_Auxiliary_logger(YTDLPLogger): 703 def __init__(self, logger: logging.Logger | None = None) -> None: 704 if logger is None: 705 logger = logging.getLogger() 706 super().__init__(logger=logger) 707 708 def prefix(self, msg: str): 709 if msg.startswith("[live-chat] "): 710 return msg 711 return f"[live-chat] {msg}" 712 713 def debug(self, msg) -> None: 714 super().debug(self.prefix(msg)) 715 716 def info(self, msg) -> None: 717 super().info(self.prefix(msg)) 718 719 def warning(self, msg) -> None: 720 super().warning(self.prefix(msg)) 721 722 def error(self, msg) -> None: 723 super().error(self.prefix(msg)) 724 725 ydl_opts = { 726 "skip_download": True, 727 "quiet": True, 728 # 'cookiefile': options.get('cookies', None), 729 "writeinfojson": options.get("write_info_json", False), 730 "writedescription": options.get("write_description", False), 731 "writethumbnail": (options.get("write_thumbnail", False) or options.get("embed_thumbnail", False)), 732 "outtmpl": base_output, 733 "retries": 10, 734 "logger": YTDLPLogger(logger=logger), 735 } 736 if options.get("proxy", None) is not None: 737 ydl_opts["proxy"] = next(iter((options.get("proxy", None) or {}).values()), None) 738 739 with YoutubeDL(ydl_opts) as ydl: 740 # base_name = ydl.prepare_filename(info_dict) 741 # result = ydl.download_with_info_file(info_dict) 742 743 if ydl._write_info_json("video", info_dict, ydl.prepare_filename(info_dict, "infojson")) or os.path.exists( 744 ydl.prepare_filename(info_dict, "infojson") 745 ): 746 created_files["info_json"] = FileInfo( 747 ydl.prepare_filename(info_dict, "infojson"), file_type="info_json" 748 ) 749 try: 750 if options.get("remove_ip_from_json"): 751 self.replace_ip_in_json(created_files["info_json"].absolute()) 752 if options.get("clean_urls"): 753 self.remove_urls_from_json(created_files["info_json"].absolute()) 754 except Exception as e: 755 self.logger.exception(str(e)) 756 757 if ydl._write_description( 758 "video", info_dict, ydl.prepare_filename(info_dict, "description") 759 ) or os.path.exists(ydl.prepare_filename(info_dict, "description")): 760 created_files["description"] = FileInfo( 761 ydl.prepare_filename(info_dict, "description"), file_type="description" 762 ) 763 764 thumbnails = ydl._write_thumbnails("video", info_dict, ydl.prepare_filename(info_dict, "thumbnail")) 765 766 if thumbnails: 767 created_files["thumbnail"] = FileInfo(thumbnails[0][0], file_type="thumbnail") 768 769 return created_files, "auxiliary" 770 771 def create_mp4(self, file_names, info_dict, options): 772 # self.logger.debug("Files: {0}".format(json.dumps(file_names))) 773 stream_manifests = list(self.file_names["streams"].items()) 774 for manifest, stream in stream_manifests: 775 index = 0 776 thumbnail = None 777 video = None 778 ext = options.get("ext", None) 779 780 ffmpeg_builder = ["ffmpeg", "-y", "-hide_banner", "-nostdin", "-loglevel", "error", "-stats"] 781 782 # Add input files 783 if stream.get("video", None): 784 input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("video").absolute())] 785 ffmpeg_builder.extend(input) 786 video = index 787 index += 1 788 789 if stream.get("audio", None): 790 input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("audio").absolute())] 791 ffmpeg_builder.extend(input) 792 index += 1 793 if video is None and ext is None: 794 ext = ".mka" 795 796 # Determine output path 797 filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id") 798 799 base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename 800 801 if len(stream_manifests) > 1: 802 base_output = f"{base_output}.{manifest}" 803 804 if ext is None: 805 self.logger.debug("No extension detected, switching to yt-dlp decided video (defaulting to MP4)") 806 ext = info_dict.get("ext", ".mp4") 807 if ext is not None and not str(ext).startswith("."): 808 ext = "." + str(ext) 809 if not base_output.endswith(ext): 810 base_output = base_output + ext 811 812 if file_names.get("thumbnail", None) and options.get("embed_thumbnail", True): 813 if file_names.get("thumbnail").exists(): 814 # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version 815 guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type) 816 817 mime_type, _ = guess(file_names.get("thumbnail")) 818 819 # If not jpeg or png, convert to png 820 if str(mime_type) not in ("image/jpeg", "image/png"): 821 self.logger.info( 822 "{} is not a JPG or PNG file, converting to png".format(file_names.get("thumbnail").name) 823 ) 824 png_thumbnail = file_names.get("thumbnail").with_suffix(".png") 825 826 thumbnail_conversion = [ 827 "ffmpeg", 828 "-y", 829 "-hide_banner", 830 "-nostdin", 831 "-loglevel", 832 "error", 833 "-stats", 834 "-i", 835 str(file_names.get("thumbnail").absolute()), 836 "-c", 837 "png", 838 "-compression_level", 839 "9", 840 "-pred", 841 "mixed", 842 str(png_thumbnail.absolute()), 843 ] 844 try: 845 result = subprocess.run( 846 thumbnail_conversion, 847 capture_output=True, 848 text=True, 849 encoding="utf-8", 850 check=True, 851 ) 852 self.logger.debug("Replacing thumbnail with .png version") 853 file_names.pop("thumbnail").unlink(missing_ok=True) 854 file_names["thumbnail"] = FileInfo(png_thumbnail, file_type="thumbnail") 855 856 except subprocess.CalledProcessError as e: 857 self.logger.exception(e.stderr) 858 self.logger.critical(e) 859 860 thumbnail = index 861 if ext.lower() != ".mkv": # Don't add input file for mkv, use attach later 862 input = [ 863 "-thread_queue_size", 864 "1024", 865 "-seekable", 866 "0", 867 "-i", 868 str(file_names.get("thumbnail").absolute()), 869 ] 870 ffmpeg_builder.extend(input) 871 index += 1 872 else: 873 self.logger.error( 874 "Thumbnail file: {} is missing, continuing without embedding".format( 875 file_names.get("thumbnail").absolute() 876 ) 877 ) 878 879 # Add faststart 880 ffmpeg_builder.extend(["-movflags", "faststart"]) 881 882 # Add mappings 883 for i in range(index): 884 input = ["-map", str(i)] 885 ffmpeg_builder.extend(input) 886 887 if thumbnail is not None: 888 if ext.lower() == ".mkv": # If file will be mkv, attach file instead 889 # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version 890 guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type) 891 mime_type, _ = guess(file_names.get("thumbnail")) 892 ffmpeg_builder.extend( 893 [ 894 "-attach", 895 str(file_names.get("thumbnail").absolute()), 896 "-metadata:s:t:0", 897 "filename=cover{}".format(file_names.get("thumbnail").suffix), 898 "-metadata:s:t:0", 899 "mimetype={}".format(mime_type or "application/octet-stream"), 900 ] 901 ) 902 903 else: # For other formats, attach using disposition instead 904 ffmpeg_builder.extend([f"-disposition:{thumbnail}", "attached_pic"]) 905 906 # Add Copy codec 907 ffmpeg_builder.extend(["-c", "copy"]) 908 909 # Add metadata 910 ffmpeg_builder.extend(["-metadata", "DATE={}".format(info_dict.get("upload_date"))]) 911 ffmpeg_builder.extend( 912 [ 913 "-metadata", 914 "COMMENT={}\n{}".format(info_dict.get("original_url"), info_dict.get("description", "")), 915 ] 916 ) 917 ffmpeg_builder.extend(["-metadata", "TITLE={}".format(info_dict.get("fulltitle"))]) 918 ffmpeg_builder.extend(["-metadata", "ARTIST={}".format(info_dict.get("channel"))]) 919 920 # Add output file to ffmpeg command 921 ffmpeg_builder.append(str(Path(base_output).absolute())) 922 923 if options.get("write_ffmpeg_command", True): 924 ffmpeg_command_file = f"{filename}.ffmpeg.txt" 925 file_names["streams"][manifest]["ffmpeg_cmd"] = FileInfo( 926 self.write_ffmpeg_command(ffmpeg_builder, ffmpeg_command_file), file_type="ffmpeg_command" 927 ) 928 929 if not (options.get("merge", True)): 930 return file_names 931 932 self.logger.debug("FFmpeg command: {}".format(" ".join(ffmpeg_builder))) 933 self.logger.info(f"Executing ffmpeg. Outputting to {ffmpeg_builder[-1]}") 934 try: 935 result = subprocess.run( 936 ffmpeg_builder, 937 capture_output=True, 938 text=True, 939 encoding="utf-8", 940 check=True, 941 ) 942 self.logger.debug(f"FFmpeg STDOUT: {result.stdout}") 943 self.logger.debug(f"FFmpeg STDERR: {result.stderr}") 944 except subprocess.CalledProcessError as e: 945 self.logger.exception(e.stderr) 946 self.logger.critical(e) 947 raise 948 949 file_names["streams"][manifest]["merged"] = FileInfo(base_output, file_type="merged") 950 self.logger.info( 951 "Successfully merged files into: {}".format(file_names["streams"][manifest].get("merged").absolute()) 952 ) 953 954 # Remove temp video and audio files 955 if not (options.get("keep_ts_files") or options.get("keep_temp_files")): 956 if file_names["streams"][manifest].get("video"): 957 self.logger.info("Removing {}".format(file_names["streams"][manifest].get("video").absolute())) 958 file_names["streams"][manifest].get("video").unlink(missing_ok=True) 959 file_names["streams"][manifest].pop("video", None) 960 if file_names["streams"][manifest].get("audio"): 961 self.logger.info("Removing {}".format(file_names["streams"][manifest].get("audio").absolute())) 962 file_names["streams"][manifest].get("audio").unlink(missing_ok=True) 963 file_names["streams"][manifest].pop("audio", None) 964 965 return file_names 966 # for file in file_names: 967 # os.remove(file) 968 969 def write_ffmpeg_command(self, command_array, filename): 970 # Determine the platform 971 """ 972 Builds a platform-compatible FFmpeg command with proper quoting. 973 974 Args: 975 command_array (list): List of arguments to append to the command. 976 filename: Filename to write command to 977 978 Returns: 979 str: A properly quoted FFmpeg command. 980 """ 981 if os.name == "nt": # Windows 982 # Handle special quoting and escaping for Windows 983 quoted_args = [] 984 for arg in command_array: 985 if "\n" in arg: 986 # Replace newlines with literal \n 987 arg = arg.replace("\n", "\\n") 988 # Escape double quotes and wrap in double quotes if necessary 989 if " " in arg or any(ch in arg for ch in ("&", "^", "%", "$", "#", '"')): 990 arg = '"{}"'.format(arg.replace('"', '\\"')) 991 quoted_args.append(arg) 992 command_string = "{}".format(" ".join(quoted_args)) 993 else: # POSIX (Linux/macOS) 994 # Use shlex.quote for safe quoting 995 # quoted_args = [shlex.quote(arg) for arg in arguments] 996 command_string = shlex.join(command_array) 997 998 with open(filename, "w", encoding="utf-8") as f: 999 f.write(command_string + "\n") 1000 1001 return filename 1002 1003 def convert_bytes(self, bytes) -> str: 1004 # List of units in order 1005 units = ["B", "KB", "MB", "GB", "TB", "PB", "EB"] 1006 1007 # Start with bytes and convert to larger units 1008 unit_index = 0 1009 while bytes >= 1024 and unit_index < len(units) - 1: 1010 bytes /= 1024 1011 unit_index += 1 1012 1013 # Format and return the result 1014 return f"{bytes:.2f} {units[unit_index]}" 1015 1016 def print_stats(self, options) -> None: 1017 if options.get("stats_as_json", False): 1018 # \033[K clears the line after printing the JSON 1019 return 1020 1021 if self.logger.getEffectiveLevel() > logging.INFO: 1022 return 1023 1024 if not (self.stats.get("video") or self.stats.get("audio")): 1025 return 1026 1027 # Build the output parts in a list 1028 parts = [f"{self.stats.get('id')}:"] 1029 1030 if self.stats.get("video"): 1031 v = self.stats.get("video", {}) 1032 v_str = f"Video: {v.get('downloaded_segments', 0)}/{v.get('latest_sequence', 0)}" 1033 if v.get("status"): 1034 v_str += f" ({v.get('status').capitalize()})" 1035 parts.append(v_str) 1036 1037 if self.stats.get("audio"): 1038 a = self.stats.get("audio", {}) 1039 a_str = f"Audio: {a.get('downloaded_segments', 0)}/{a.get('latest_sequence', 0)}" 1040 # Note: Fixed a likely typo in your original code where you checked 1041 # video status while printing audio stats 1042 if a.get("status"): 1043 a_str += f" ({a.get('status').capitalize()})" 1044 parts.append(a_str) 1045 1046 if self.stats.get("video", {}).get("current_filesize") or self.stats.get("audio", {}).get("current_filesize"): 1047 size = self.stats.get("video", {}).get("current_filesize", 0) + self.stats.get("audio", {}).get( 1048 "current_filesize", 0 1049 ) 1050 parts.append(f"~{self.convert_bytes(size)} downloaded") 1051 1052 # Join everything with commas or spaces 1053 " ".join(parts) 1054 1055 if options.get("new_line", False): 1056 pass 1057 else: 1058 # \r moves to start, \033[K clears anything left over from the previous longer line 1059 pass 1060 1061 def add_url_param(self, url: str, key, value) -> str: 1062 parsed = urlparse(url) 1063 query = parse_qs(parsed.query) 1064 query[key] = [value] # add or replace parameter 1065 1066 new_query = urlencode(query, doseq=True) 1067 new_url = parsed._replace(query=new_query) 1068 return str(urlunparse(new_url)) 1069 1070 def trim_info_json(self, info_dict: dict, keys_to_keep: set): 1071 return {k: info_dict[k] for k in info_dict.keys() & keys_to_keep} 1072 1073 def remove_format_segment_playlist_from_info_dict(self, info_dict: dict) -> None: 1074 for stream_format in info_dict.get("formats", []): 1075 with contextlib.suppress(KeyError, AttributeError, TypeError): 1076 stream_format.pop("fragments", None) 1077 1078 def refresh_info_json( 1079 self, 1080 update_threshold: float, 1081 video_id, 1082 cookies=None, 1083 additional_options=None, 1084 include_dash=False, 1085 include_m3u8=False, 1086 ): 1087 # Check if time difference is greater than the threshold. If doesn't exist, subtraction of zero will always be true 1088 if time.time() - self.refresh_json.get("refresh_time", 0.0) > update_threshold: 1089 self.refresh_json, self.live_status = get_video_info( 1090 video_id=video_id, 1091 wait=False, 1092 cookies=cookies, 1093 additional_options=additional_options, 1094 include_dash=include_dash, 1095 include_m3u8=include_m3u8, 1096 clean_info_dict=True, 1097 ) 1098 1099 # Remove unnecessary items for info.json used purely for url refresh 1100 self.remove_format_segment_playlist_from_info_dict(self.refresh_json) 1101 1102 self.refresh_json.pop("thumbnails", None) 1103 self.refresh_json.pop("tags", None) 1104 self.refresh_json.pop("description", None) 1105 1106 # Add refresh time for reference 1107 self.refresh_json["refresh_time"] = time.time() 1108 1109 return self.refresh_json, self.live_status 1110 return self.refresh_json, self.live_status
40 def __init__( 41 self, 42 kill_all: threading.Event = threading.Event(), 43 cleanup: threading.Event = threading.Event(), 44 logger: logging.Logger | logging.LoggerAdapter | None = None, 45 kill_this: threading.Event | None = None, 46 ) -> None: 47 if logger: 48 self.logger = logger 49 else: 50 # 1. Create a Named Logger instance and assign it to self.logger 51 self.logger = logging.getLogger(self.__class__.__name__) 52 # Ensure it processes messages (important if you don't call setup_logging immediately) 53 if isinstance(self.logger, logging.Logger): 54 self.logger.propagate = False 55 self.kill_this = kill_this or threading.Event() 56 57 # Global state converted to instance attributes 58 self.kill_all = kill_all 59 self.cleanup = cleanup 60 self.live_chat_result = None 61 self.chat_timeout = None 62 # File name dictionary 63 self.file_names = {"databases": [], "streams": {}} 64 self.stats = {} 65 66 self.refresh_json = {} 67 self.live_status = "" 68 self.lock: threading.Lock = threading.Lock()
71 def download_stream( 72 self, 73 info_dict, 74 resolution, 75 batch_size=5, 76 max_workers=1, 77 folder=None, 78 file_name=None, 79 keep_database=False, 80 cookies=None, 81 retries=5, 82 yt_dlp_options=None, 83 proxies=None, 84 yt_dlp_sort=None, 85 include_dash=False, 86 include_m3u8=False, 87 force_m3u8=False, 88 manifest=0, 89 **kwargs, 90 ): 91 try: 92 download_params = locals().copy() 93 download_params.update({"download_function": self.download_stream}) 94 file = None 95 filetype = None 96 97 with DownloadStream( 98 info_dict, 99 resolution=resolution, 100 batch_size=batch_size, 101 max_workers=max_workers, 102 folder=folder, 103 file_name=file_name, 104 cookies=cookies, 105 fragment_retries=retries, 106 yt_dlp_options=yt_dlp_options, 107 proxies=proxies, 108 yt_dlp_sort=yt_dlp_sort, 109 include_dash=include_dash, 110 include_m3u8=include_m3u8, 111 force_m3u8=force_m3u8, 112 download_params=download_params, 113 livestream_coordinator=self, 114 **kwargs, 115 ) as downloader: 116 self.stats["status"] = "Recording" 117 downloader.live_dl() 118 file_name = downloader.combine_segments_to_file(downloader.merged_file_name) 119 if not keep_database: 120 self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}") 121 downloader.delete_temp_database() 122 elif downloader.temp_db_file != ":memory:": 123 database_file = FileInfo(downloader.temp_db_file, file_type="database", format=downloader.format) 124 self.file_names["databases"].append(database_file) 125 126 file = FileInfo(file_name, file_type=downloader.type, format=downloader.format) 127 filetype = downloader.type 128 129 self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file}) 130 131 return file, filetype 132 except Exception: 133 self.logger.exception("Unexpected error occurred while downloading stream") 134 raise
137 def download_stream_direct( 138 self, 139 info_dict, 140 resolution, 141 batch_size, 142 max_workers, 143 folder=None, 144 file_name=None, 145 keep_state=False, 146 cookies=None, 147 retries=5, 148 yt_dlp_options=None, 149 proxies=None, 150 yt_dlp_sort=None, 151 include_dash=False, 152 include_m3u8=False, 153 force_m3u8=False, 154 manifest=0, 155 **kwargs, 156 ): 157 try: 158 download_params = locals().copy() 159 download_params.update({"download_function": self.download_stream_direct}) 160 file = None 161 filetype = None 162 163 with DownloadStreamDirect( 164 info_dict, 165 resolution=resolution, 166 max_workers=max_workers, 167 folder=folder, 168 file_name=file_name, 169 cookies=cookies, 170 fragment_retries=retries, 171 yt_dlp_options=yt_dlp_options, 172 proxies=proxies, 173 yt_dlp_sort=yt_dlp_sort, 174 include_dash=include_dash, 175 include_m3u8=include_m3u8, 176 force_m3u8=force_m3u8, 177 download_params=download_params, 178 livestream_coordinator=self, 179 **kwargs, 180 ) as downloader: 181 self.stats["status"] = "Recording" 182 file_name = downloader.live_dl() 183 file = FileInfo(file_name, file_type=downloader.type, format=downloader.format) 184 filetype = downloader.type 185 downloader.delete_state_file() 186 187 self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file}) 188 return file, filetype 189 except Exception: 190 self.logger.exception("Unexpected error occurred while downloading stream") 191 raise
193 def recover_stream( 194 self, 195 info_dict, 196 resolution, 197 batch_size=5, 198 max_workers=5, 199 folder=None, 200 file_name=None, 201 keep_database=False, 202 cookies=None, 203 retries=5, 204 yt_dlp_options=None, 205 proxies=None, 206 yt_dlp_sort=None, 207 force_merge=False, 208 recovery_failure_tolerance=0, 209 manifest=0, 210 stream_urls: list | None = None, 211 no_merge=False, 212 **kwargs, 213 ): 214 if stream_urls is None: 215 stream_urls = [] 216 try: 217 file = None 218 filetype = None 219 220 with StreamRecovery( 221 info_dict, 222 resolution=resolution, 223 batch_size=batch_size, 224 max_workers=max_workers, 225 folder=folder, 226 file_name=file_name, 227 cookies=cookies, 228 fragment_retries=retries, 229 proxies=proxies, 230 yt_dlp_sort=yt_dlp_sort, 231 livestream_coordinator=self, 232 stream_urls=stream_urls, 233 **kwargs, 234 ) as downloader: 235 self.stats["status"] = "Recording" 236 result = downloader.live_dl() 237 if (force_merge or result <= 0 or result <= recovery_failure_tolerance) and (not no_merge): 238 if result > 0: 239 self.logger.warning( 240 f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting" 241 ) 242 file_name = downloader.combine_segments_to_file(downloader.merged_file_name) 243 if not keep_database: 244 self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}") 245 downloader.delete_temp_database() 246 elif downloader.temp_db_file != ":memory:": 247 database_file = FileInfo( 248 downloader.temp_db_file, file_type="database", format=downloader.format 249 ) 250 self.file_names["databases"].append(database_file) 251 else: 252 msg = f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting" 253 raise VideoDownloadError(msg) 254 file = FileInfo(file_name, file_type=downloader.type, format=downloader.format) 255 filetype = downloader.type 256 257 self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file}) 258 except Exception: 259 self.logger.exception("Unexpected error occurred while downloading stream") 260 raise 261 return file, filetype
263 def submit_download( 264 self, executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False 265 ): 266 extra_kwargs = {} 267 268 # Options only not used by recovery 269 if not options.get("recovery", False): 270 extra_kwargs.update( 271 { 272 "include_dash": options.get("dash", False), 273 "include_m3u8": options.get("m3u8", False), 274 "force_m3u8": options.get("force_m3u8", False), 275 } 276 ) 277 278 # Select the function 279 if options.get("recovery", False): 280 func = self.recover_stream 281 extra_kwargs.update( 282 { 283 "force_merge": options.get("force_recovery_merge", False), 284 "recovery_failure_tolerance": options.get("recovery_failure_tolerance", 0), 285 } 286 ) 287 elif options.get("direct_to_ts", False): 288 func = self.download_stream_direct 289 extra_kwargs.update( 290 { 291 "keep_state": options.get("keep_temp_files", False) or options.get("keep_database_file", False), 292 } 293 ) 294 else: 295 func = self.download_stream 296 extra_kwargs.update( 297 { 298 "keep_database": options.get("keep_temp_files", False) or options.get("keep_database_file", False), 299 } 300 ) 301 302 kwargs = { 303 "info_dict": info_dict, 304 "resolution": "audio_only" if is_audio else resolution, 305 "batch_size": options.get("batch_size", 1), 306 "max_workers": options.get("threads", 1), 307 "folder": download_folder, 308 "file_name": file_name, 309 "retries": options.get("segment_retries"), 310 "cookies": options.get("cookies"), 311 "yt_dlp_options": options.get("ytdlp_options", None), 312 "proxies": options.get("proxy", None), 313 "yt_dlp_sort": options.get("custom_sort", None), 314 } 315 316 if extra_kwargs: 317 kwargs.update(extra_kwargs) 318 319 # Add any extra values not yet existing 320 for key, value in options.items(): 321 kwargs.setdefault(key, value) 322 323 self.logger.debug(f"Starting executor with: {json.dumps(kwargs)}") 324 325 # Submit to executor 326 future = executor.submit(func, **kwargs) 327 futures.add(future) 328 return future
331 def download_segments( 332 self, info_dict, resolution="best", options=None, thread_event: threading.Event | None = None 333 ) -> None: 334 if options is None: 335 options = {} 336 futures = set() 337 338 if thread_event is not None: 339 self.kill_all = thread_event 340 341 self.stats["id"] = info_dict.get("id", None) 342 self.stats["status"] = "Starting" 343 self.logger.debug(json.dumps(options, indent=4)) 344 outputFile = self.output_filename(info_dict=info_dict, outtmpl=options.get("output")) 345 file_name = None 346 # Requires testing 347 if options.get("temp_folder") is not None and options.get("temp_folder") != os.path.dirname(outputFile): 348 _output_folder, file_name = os.path.split(outputFile) 349 download_folder = self.output_filename(info_dict=info_dict, outtmpl=options.get("temp_folder")) 350 options["temp_folder"] = download_folder 351 else: 352 download_folder, file_name = os.path.split(outputFile) 353 options["filename"] = file_name 354 if download_folder: 355 os.makedirs(download_folder, exist_ok=True) 356 357 with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor: 358 done = not_done = set() 359 try: 360 # Download auxiliary files (thumbnail, info,json etc) 361 auxiliary_thread = executor.submit(self.download_auxiliary_files, info_dict=info_dict, options=options) 362 futures.add(auxiliary_thread) 363 364 live_chat_thread = None 365 if options.get("live_chat", False) is True: 366 live_chat_thread = threading.Thread( 367 target=self.download_live_chat, args=(info_dict, options), daemon=True 368 ) 369 live_chat_thread.start() 370 371 format_check = Formats().getFormatURL( 372 info_json=info_dict, 373 resolution=resolution, 374 sort=options.get("custom_sort"), 375 include_dash=(options.get("dash", False) and not options.get("recovery", False)), 376 include_m3u8=options.get("m3u8", False), 377 force_m3u8=options.get("force_m3u8", False), 378 ) 379 if not format_check: 380 msg = "Resolution is not valid or does not exist in stream" 381 raise ValueError(msg) 382 # For use of specificed format. Expects two values, but can work with more 383 # Video + Audio 384 385 if resolution.lower() not in ("audio_only", "ba"): 386 # Video 387 self.submit_download( 388 executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False 389 ) 390 if format_check.protocol != "m3u8_native": 391 # Audio 392 self.submit_download( 393 executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True 394 ) 395 # Audio-only 396 else: 397 self.submit_download( 398 executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True 399 ) 400 401 while True: 402 if self.kill_all.is_set() or self.kill_this.is_set(): 403 msg = "Thread kill event is set, ending..." 404 raise KeyboardInterrupt(msg) 405 done, not_done = concurrent.futures.wait( 406 futures, timeout=1, return_when=concurrent.futures.FIRST_COMPLETED 407 ) 408 # Continuously check for completion or interruption 409 for future in done: 410 result, type = future.result() 411 self.logger.info(f"\033[31m{result}\033[0m") 412 413 if type == "auxiliary": 414 self.file_names.update(result) 415 416 # Remove parts that would not be needed by stream downloaders 417 self.remove_format_segment_playlist_from_info_dict(info_dict=info_dict) 418 info_dict.pop("thumbnails", None) 419 # Remove no longer required parts for the rest of the downloader 420 # info_dict = self.trim_info_json(info_dict=info_dict, keys_to_keep={"id", "ext", "upload_date", "original_url", "description", "fulltitle", "channel"}) 421 422 elif str(type).lower() == "video": 423 # self.file_names['video'] = result 424 pass 425 elif str(type).lower() == "audio": 426 # self.file_names['audio'] = result 427 pass 428 elif str(type).lower() == "live_chat": 429 pass 430 else: 431 self.file_names[str(type)] = result 432 433 futures.discard(future) 434 435 if len(not_done) <= 0: 436 break 437 # else: 438 # time.sleep(0.9) 439 self.print_stats(options=options) 440 441 if live_chat_thread and live_chat_thread.is_alive(): 442 self.logger.info("Media downloads finished, waiting for live chat to end...") 443 if options.get("stop_chat_when_done"): 444 self.chat_timeout = time.time() 445 while live_chat_thread.is_alive() and not (self.kill_all.is_set() or self.kill_this.is_set()): 446 time.sleep(0.1) 447 448 except KeyboardInterrupt: 449 self.kill_this.set() 450 self.logger.debug("Keyboard interrupt detected") 451 if len(not_done) > 0: 452 self.logger.debug("Cancelling remaining threads") 453 for future in not_done: 454 _ = future.cancel() 455 done, not_done = concurrent.futures.wait( 456 futures, timeout=5, return_when=concurrent.futures.ALL_COMPLETED 457 ) 458 executor.shutdown(wait=False, cancel_futures=True) 459 self.logger.debug("Shutdown all threads") 460 self.stats["status"] = "Cancelled" 461 raise 462 463 self.stats["status"] = "Muxing" 464 self.create_mp4(file_names=self.file_names, info_dict=info_dict, options=options) 465 466 self.stats["status"] = "Moving" 467 self.move_to_final(options=options, output_file=outputFile, file_names=self.file_names) 468 self.stats["status"] = "Finished" 469 # move_to_final(info_dict, options, file_names)
476 def move_to_final(self, options, output_file, file_names) -> None: 477 def maybe_move(key, dest_func, file_names_dict=file_names, option_flag=None) -> None: 478 """key: key in file_names 479 dest_func: func -> pathlib -> string dest path 480 delete_if_false: if True, delete when option_flag not set 481 option_flag: name of boolean option (write_thumbnail etc) 482 """ 483 f = file_names_dict.get(key) 484 if not f: 485 return 486 try: 487 # deletion case (thumbnail / ffmpeg) 488 if option_flag is not None and not options.get(option_flag, False): 489 self.logger.info(f"Removing {f.absolute()}") 490 f.unlink(missing_ok=True) 491 file_names_dict.pop(key, None) 492 return 493 494 dest = dest_func(f) 495 if str(f).strip() != str(dest).strip(): 496 self.logger.info(f"Moving {f.absolute()} → {dest}") 497 shutil.move(f.absolute(), dest) 498 else: 499 self.logger.debug(f"{f.absolute()} is already in final destination") 500 except Exception as e: 501 self.logger.exception(f"unable to move {key}: {e}") 502 503 # ensure output dir exists 504 out_dir = os.path.dirname(output_file) 505 if out_dir: 506 os.makedirs(out_dir, exist_ok=True) 507 508 # === individual file handlers === 509 510 maybe_move("thumbnail", lambda f: f"{output_file}{f.suffix}", option_flag="write_thumbnail") 511 512 maybe_move( 513 "info_json", 514 lambda f: f"{output_file}.info.json", 515 ) 516 517 maybe_move( 518 "description", 519 lambda f: f"{output_file}{f.suffix}", 520 ) 521 522 stream_manifests = list(self.file_names["streams"].items()) 523 for manifest, stream in stream_manifests: 524 stream_output_file = output_file 525 if len(stream_manifests) > 1: 526 stream_output_file = f"{output_file}.{manifest}" 527 maybe_move("video", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream) 528 529 maybe_move("audio", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream) 530 531 maybe_move("merged", lambda f: f"{stream_output_file}{f.suffix}", file_names_dict=stream) 532 533 maybe_move( 534 "live_chat", 535 lambda f: f"{output_file}.live_chat.zip", 536 ) 537 538 # special: databases = list 539 try: 540 for f in file_names.get("databases", []): 541 dest = f"{output_file}.{f._format}{f.suffix}" 542 if str(f).strip() != str(dest).strip(): 543 self.logger.info(f"Moving {f.absolute()} → {dest}") 544 shutil.move(f.absolute(), dest) 545 except Exception as e: 546 self.logger.exception(f"unable to move database files: {e}") 547 548 # special: ffmpeg_cmd 549 maybe_move("ffmpeg_cmd", lambda f: f"{output_file}.ffmpeg.txt", option_flag="write_ffmpeg_command") 550 551 # remove temp folder 552 """ 553 if options.get('temp_folder', None) is not None: 554 try: 555 os.rmdir(options.get('temp_folder')) 556 except OSError as e: 557 if e.errno == errno.ENOTEMPTY: 558 self.logger.warning(f"Error: Directory not empty: {e.filename}") 559 else: 560 self.logger.exception(f"Error removing temp folder: {e}") 561 except Exception as e: 562 self.logger.exception(f"Error removing temp folder: {e}") 563 """ 564 self.logger.info("Finished moving files from temporary directory to output destination")
566 def download_live_chat(self, info_dict, options): 567 filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id") 568 569 base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename 570 571 class YTDLP_Chat_logger(YTDLPLogger): 572 def __init__(self, logger: logging.Logger | logging.LoggerAdapter | None = None) -> None: 573 if logger is None: 574 logger = logging.getLogger() 575 super().__init__(logger=logger) 576 577 def prefix(self, msg: str): 578 if msg.startswith("[live-chat] "): 579 return msg 580 return f"[live-chat] {msg}" 581 582 def debug(self, msg) -> None: 583 super().debug(self.prefix(msg)) 584 585 def info(self, msg) -> None: 586 super().info(self.prefix(msg)) 587 588 def warning(self, msg) -> None: 589 super().warning(self.prefix(msg)) 590 591 def error(self, msg) -> None: 592 super().error(self.prefix(msg)) 593 594 logger = self.logger 595 596 ydl_opts = { 597 "skip_download": True, # Skip downloading video/audio 598 "logger": YTDLP_Chat_logger(logger=logger), 599 #'quiet': True, 600 "cookiefile": options.get("cookies", None), 601 "retries": 25, 602 "concurrent_fragment_downloads": 3, 603 #'live_from_start': True, 604 "writesubtitles": True, # Extract subtitles (live chat) 605 "subtitlesformat": "json", # Set format to JSON 606 "subtitleslangs": ["live_chat"], # Only extract live chat subtitles 607 "outtmpl": base_output, # Save to a JSON file 608 } 609 self.logger.debug(options.get("ytdlp_options", {})) 610 ydl_opts.update(options.get("ytdlp_options", {})) 611 612 livechat_filename = base_output + ".live_chat.json" 613 614 self.logger.info(f"Downloading live chat to: {livechat_filename}") 615 616 # Download live chat using yt-dlp 617 with YoutubeDL(ydl_opts) as ydl: 618 try: 619 ydl.download([info_dict.get("original_url", info_dict.get("webpage_url", info_dict.get("url")))]) 620 except Exception as e: 621 self.logger.exception(f"Failed to download live chat: {e}") 622 623 time.sleep(1) 624 part_file = f"{livechat_filename}.part" 625 if os.path.exists(part_file): 626 # Append part to 627 chunk_size = 1024 * 1024 * 10 # number of characters per chunk, up to 10M characters 628 with open(part_file, encoding="utf-8") as fa, open(livechat_filename, "a", encoding="utf-8") as fb: 629 while chunk := fa.read(chunk_size): 630 fb.write(chunk) 631 os.remove(f"{livechat_filename}.part") 632 633 try: 634 if not os.path.exists(livechat_filename): 635 self.logger.warning(f"Live chat file not found: {livechat_filename}") 636 return None, None 637 638 zip_filename = base_output + ".live_chat.zip" 639 with zipfile.ZipFile( 640 zip_filename, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=9, allowZip64=True 641 ) as zipf: 642 zipf.write(livechat_filename, arcname=os.path.basename(livechat_filename)) 643 os.remove(livechat_filename) 644 live_chat = {"live_chat": FileInfo(zip_filename, file_type="live_chat")} 645 self.file_names.update(live_chat) 646 return live_chat, "live_chat" 647 except Exception as e: 648 self.logger.exception(f"\033[31m{e}\033[0m")
650 def replace_ip_in_json(self, file_name) -> None: 651 pattern = re.compile(r"((?:[0-9]{1,3}\.){3}[0-9]{1,3})|((?:[a-f0-9]{1,4}:){7}[a-f0-9]{1,4})") 652 653 with open(file_name, encoding="utf-8") as file: 654 content = file.read() 655 656 modified_content = re.sub(pattern, "0.0.0.0", content) 657 658 with open(file_name, "w", encoding="utf-8") as file: 659 file.write(modified_content)
661 def remove_urls_from_json(self, file_name) -> None: 662 with open(file_name, encoding="utf-8") as file: 663 data = json.load(file) 664 665 if data.get("formats", None) is not None: 666 for format in data["formats"]: 667 if format.get("url") is not None: 668 format["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 669 670 if format.get("manifest_url") is not None: 671 format["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 672 673 format.pop("fragment_base_url", None) 674 format.pop("fragments", None) 675 676 if data.get("thumbnails", None) is not None: 677 for thumbnail in data["thumbnails"]: 678 if thumbnail.get("url", None) is not None: 679 parsed_url = urlparse(thumbnail.get("url", "")) 680 thumbnail["url"] = f"{parsed_url.scheme}://{parsed_url.netloc}{parsed_url.path}" 681 682 if data.get("url", None) is not None: 683 data["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 684 685 if data.get("manifest_url", None) is not None: 686 data["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", "")) 687 688 data["removed_urls"] = True 689 690 with open(file_name, "w", encoding="utf-8") as file: 691 json.dump(data, file)
693 def download_auxiliary_files(self, info_dict, options): 694 filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id") 695 696 base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename 697 698 created_files = {} 699 700 logger = self.logger 701 702 class YTDLP_Auxiliary_logger(YTDLPLogger): 703 def __init__(self, logger: logging.Logger | None = None) -> None: 704 if logger is None: 705 logger = logging.getLogger() 706 super().__init__(logger=logger) 707 708 def prefix(self, msg: str): 709 if msg.startswith("[live-chat] "): 710 return msg 711 return f"[live-chat] {msg}" 712 713 def debug(self, msg) -> None: 714 super().debug(self.prefix(msg)) 715 716 def info(self, msg) -> None: 717 super().info(self.prefix(msg)) 718 719 def warning(self, msg) -> None: 720 super().warning(self.prefix(msg)) 721 722 def error(self, msg) -> None: 723 super().error(self.prefix(msg)) 724 725 ydl_opts = { 726 "skip_download": True, 727 "quiet": True, 728 # 'cookiefile': options.get('cookies', None), 729 "writeinfojson": options.get("write_info_json", False), 730 "writedescription": options.get("write_description", False), 731 "writethumbnail": (options.get("write_thumbnail", False) or options.get("embed_thumbnail", False)), 732 "outtmpl": base_output, 733 "retries": 10, 734 "logger": YTDLPLogger(logger=logger), 735 } 736 if options.get("proxy", None) is not None: 737 ydl_opts["proxy"] = next(iter((options.get("proxy", None) or {}).values()), None) 738 739 with YoutubeDL(ydl_opts) as ydl: 740 # base_name = ydl.prepare_filename(info_dict) 741 # result = ydl.download_with_info_file(info_dict) 742 743 if ydl._write_info_json("video", info_dict, ydl.prepare_filename(info_dict, "infojson")) or os.path.exists( 744 ydl.prepare_filename(info_dict, "infojson") 745 ): 746 created_files["info_json"] = FileInfo( 747 ydl.prepare_filename(info_dict, "infojson"), file_type="info_json" 748 ) 749 try: 750 if options.get("remove_ip_from_json"): 751 self.replace_ip_in_json(created_files["info_json"].absolute()) 752 if options.get("clean_urls"): 753 self.remove_urls_from_json(created_files["info_json"].absolute()) 754 except Exception as e: 755 self.logger.exception(str(e)) 756 757 if ydl._write_description( 758 "video", info_dict, ydl.prepare_filename(info_dict, "description") 759 ) or os.path.exists(ydl.prepare_filename(info_dict, "description")): 760 created_files["description"] = FileInfo( 761 ydl.prepare_filename(info_dict, "description"), file_type="description" 762 ) 763 764 thumbnails = ydl._write_thumbnails("video", info_dict, ydl.prepare_filename(info_dict, "thumbnail")) 765 766 if thumbnails: 767 created_files["thumbnail"] = FileInfo(thumbnails[0][0], file_type="thumbnail") 768 769 return created_files, "auxiliary"
771 def create_mp4(self, file_names, info_dict, options): 772 # self.logger.debug("Files: {0}".format(json.dumps(file_names))) 773 stream_manifests = list(self.file_names["streams"].items()) 774 for manifest, stream in stream_manifests: 775 index = 0 776 thumbnail = None 777 video = None 778 ext = options.get("ext", None) 779 780 ffmpeg_builder = ["ffmpeg", "-y", "-hide_banner", "-nostdin", "-loglevel", "error", "-stats"] 781 782 # Add input files 783 if stream.get("video", None): 784 input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("video").absolute())] 785 ffmpeg_builder.extend(input) 786 video = index 787 index += 1 788 789 if stream.get("audio", None): 790 input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("audio").absolute())] 791 ffmpeg_builder.extend(input) 792 index += 1 793 if video is None and ext is None: 794 ext = ".mka" 795 796 # Determine output path 797 filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id") 798 799 base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename 800 801 if len(stream_manifests) > 1: 802 base_output = f"{base_output}.{manifest}" 803 804 if ext is None: 805 self.logger.debug("No extension detected, switching to yt-dlp decided video (defaulting to MP4)") 806 ext = info_dict.get("ext", ".mp4") 807 if ext is not None and not str(ext).startswith("."): 808 ext = "." + str(ext) 809 if not base_output.endswith(ext): 810 base_output = base_output + ext 811 812 if file_names.get("thumbnail", None) and options.get("embed_thumbnail", True): 813 if file_names.get("thumbnail").exists(): 814 # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version 815 guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type) 816 817 mime_type, _ = guess(file_names.get("thumbnail")) 818 819 # If not jpeg or png, convert to png 820 if str(mime_type) not in ("image/jpeg", "image/png"): 821 self.logger.info( 822 "{} is not a JPG or PNG file, converting to png".format(file_names.get("thumbnail").name) 823 ) 824 png_thumbnail = file_names.get("thumbnail").with_suffix(".png") 825 826 thumbnail_conversion = [ 827 "ffmpeg", 828 "-y", 829 "-hide_banner", 830 "-nostdin", 831 "-loglevel", 832 "error", 833 "-stats", 834 "-i", 835 str(file_names.get("thumbnail").absolute()), 836 "-c", 837 "png", 838 "-compression_level", 839 "9", 840 "-pred", 841 "mixed", 842 str(png_thumbnail.absolute()), 843 ] 844 try: 845 result = subprocess.run( 846 thumbnail_conversion, 847 capture_output=True, 848 text=True, 849 encoding="utf-8", 850 check=True, 851 ) 852 self.logger.debug("Replacing thumbnail with .png version") 853 file_names.pop("thumbnail").unlink(missing_ok=True) 854 file_names["thumbnail"] = FileInfo(png_thumbnail, file_type="thumbnail") 855 856 except subprocess.CalledProcessError as e: 857 self.logger.exception(e.stderr) 858 self.logger.critical(e) 859 860 thumbnail = index 861 if ext.lower() != ".mkv": # Don't add input file for mkv, use attach later 862 input = [ 863 "-thread_queue_size", 864 "1024", 865 "-seekable", 866 "0", 867 "-i", 868 str(file_names.get("thumbnail").absolute()), 869 ] 870 ffmpeg_builder.extend(input) 871 index += 1 872 else: 873 self.logger.error( 874 "Thumbnail file: {} is missing, continuing without embedding".format( 875 file_names.get("thumbnail").absolute() 876 ) 877 ) 878 879 # Add faststart 880 ffmpeg_builder.extend(["-movflags", "faststart"]) 881 882 # Add mappings 883 for i in range(index): 884 input = ["-map", str(i)] 885 ffmpeg_builder.extend(input) 886 887 if thumbnail is not None: 888 if ext.lower() == ".mkv": # If file will be mkv, attach file instead 889 # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version 890 guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type) 891 mime_type, _ = guess(file_names.get("thumbnail")) 892 ffmpeg_builder.extend( 893 [ 894 "-attach", 895 str(file_names.get("thumbnail").absolute()), 896 "-metadata:s:t:0", 897 "filename=cover{}".format(file_names.get("thumbnail").suffix), 898 "-metadata:s:t:0", 899 "mimetype={}".format(mime_type or "application/octet-stream"), 900 ] 901 ) 902 903 else: # For other formats, attach using disposition instead 904 ffmpeg_builder.extend([f"-disposition:{thumbnail}", "attached_pic"]) 905 906 # Add Copy codec 907 ffmpeg_builder.extend(["-c", "copy"]) 908 909 # Add metadata 910 ffmpeg_builder.extend(["-metadata", "DATE={}".format(info_dict.get("upload_date"))]) 911 ffmpeg_builder.extend( 912 [ 913 "-metadata", 914 "COMMENT={}\n{}".format(info_dict.get("original_url"), info_dict.get("description", "")), 915 ] 916 ) 917 ffmpeg_builder.extend(["-metadata", "TITLE={}".format(info_dict.get("fulltitle"))]) 918 ffmpeg_builder.extend(["-metadata", "ARTIST={}".format(info_dict.get("channel"))]) 919 920 # Add output file to ffmpeg command 921 ffmpeg_builder.append(str(Path(base_output).absolute())) 922 923 if options.get("write_ffmpeg_command", True): 924 ffmpeg_command_file = f"{filename}.ffmpeg.txt" 925 file_names["streams"][manifest]["ffmpeg_cmd"] = FileInfo( 926 self.write_ffmpeg_command(ffmpeg_builder, ffmpeg_command_file), file_type="ffmpeg_command" 927 ) 928 929 if not (options.get("merge", True)): 930 return file_names 931 932 self.logger.debug("FFmpeg command: {}".format(" ".join(ffmpeg_builder))) 933 self.logger.info(f"Executing ffmpeg. Outputting to {ffmpeg_builder[-1]}") 934 try: 935 result = subprocess.run( 936 ffmpeg_builder, 937 capture_output=True, 938 text=True, 939 encoding="utf-8", 940 check=True, 941 ) 942 self.logger.debug(f"FFmpeg STDOUT: {result.stdout}") 943 self.logger.debug(f"FFmpeg STDERR: {result.stderr}") 944 except subprocess.CalledProcessError as e: 945 self.logger.exception(e.stderr) 946 self.logger.critical(e) 947 raise 948 949 file_names["streams"][manifest]["merged"] = FileInfo(base_output, file_type="merged") 950 self.logger.info( 951 "Successfully merged files into: {}".format(file_names["streams"][manifest].get("merged").absolute()) 952 ) 953 954 # Remove temp video and audio files 955 if not (options.get("keep_ts_files") or options.get("keep_temp_files")): 956 if file_names["streams"][manifest].get("video"): 957 self.logger.info("Removing {}".format(file_names["streams"][manifest].get("video").absolute())) 958 file_names["streams"][manifest].get("video").unlink(missing_ok=True) 959 file_names["streams"][manifest].pop("video", None) 960 if file_names["streams"][manifest].get("audio"): 961 self.logger.info("Removing {}".format(file_names["streams"][manifest].get("audio").absolute())) 962 file_names["streams"][manifest].get("audio").unlink(missing_ok=True) 963 file_names["streams"][manifest].pop("audio", None) 964 965 return file_names 966 # for file in file_names: 967 # os.remove(file)
969 def write_ffmpeg_command(self, command_array, filename): 970 # Determine the platform 971 """ 972 Builds a platform-compatible FFmpeg command with proper quoting. 973 974 Args: 975 command_array (list): List of arguments to append to the command. 976 filename: Filename to write command to 977 978 Returns: 979 str: A properly quoted FFmpeg command. 980 """ 981 if os.name == "nt": # Windows 982 # Handle special quoting and escaping for Windows 983 quoted_args = [] 984 for arg in command_array: 985 if "\n" in arg: 986 # Replace newlines with literal \n 987 arg = arg.replace("\n", "\\n") 988 # Escape double quotes and wrap in double quotes if necessary 989 if " " in arg or any(ch in arg for ch in ("&", "^", "%", "$", "#", '"')): 990 arg = '"{}"'.format(arg.replace('"', '\\"')) 991 quoted_args.append(arg) 992 command_string = "{}".format(" ".join(quoted_args)) 993 else: # POSIX (Linux/macOS) 994 # Use shlex.quote for safe quoting 995 # quoted_args = [shlex.quote(arg) for arg in arguments] 996 command_string = shlex.join(command_array) 997 998 with open(filename, "w", encoding="utf-8") as f: 999 f.write(command_string + "\n") 1000 1001 return filename
Builds a platform-compatible FFmpeg command with proper quoting.
Arguments:
- command_array (list): List of arguments to append to the command.
- filename: Filename to write command to
Returns:
str: A properly quoted FFmpeg command.
1003 def convert_bytes(self, bytes) -> str: 1004 # List of units in order 1005 units = ["B", "KB", "MB", "GB", "TB", "PB", "EB"] 1006 1007 # Start with bytes and convert to larger units 1008 unit_index = 0 1009 while bytes >= 1024 and unit_index < len(units) - 1: 1010 bytes /= 1024 1011 unit_index += 1 1012 1013 # Format and return the result 1014 return f"{bytes:.2f} {units[unit_index]}"
1016 def print_stats(self, options) -> None: 1017 if options.get("stats_as_json", False): 1018 # \033[K clears the line after printing the JSON 1019 return 1020 1021 if self.logger.getEffectiveLevel() > logging.INFO: 1022 return 1023 1024 if not (self.stats.get("video") or self.stats.get("audio")): 1025 return 1026 1027 # Build the output parts in a list 1028 parts = [f"{self.stats.get('id')}:"] 1029 1030 if self.stats.get("video"): 1031 v = self.stats.get("video", {}) 1032 v_str = f"Video: {v.get('downloaded_segments', 0)}/{v.get('latest_sequence', 0)}" 1033 if v.get("status"): 1034 v_str += f" ({v.get('status').capitalize()})" 1035 parts.append(v_str) 1036 1037 if self.stats.get("audio"): 1038 a = self.stats.get("audio", {}) 1039 a_str = f"Audio: {a.get('downloaded_segments', 0)}/{a.get('latest_sequence', 0)}" 1040 # Note: Fixed a likely typo in your original code where you checked 1041 # video status while printing audio stats 1042 if a.get("status"): 1043 a_str += f" ({a.get('status').capitalize()})" 1044 parts.append(a_str) 1045 1046 if self.stats.get("video", {}).get("current_filesize") or self.stats.get("audio", {}).get("current_filesize"): 1047 size = self.stats.get("video", {}).get("current_filesize", 0) + self.stats.get("audio", {}).get( 1048 "current_filesize", 0 1049 ) 1050 parts.append(f"~{self.convert_bytes(size)} downloaded") 1051 1052 # Join everything with commas or spaces 1053 " ".join(parts) 1054 1055 if options.get("new_line", False): 1056 pass 1057 else: 1058 # \r moves to start, \033[K clears anything left over from the previous longer line 1059 pass
1061 def add_url_param(self, url: str, key, value) -> str: 1062 parsed = urlparse(url) 1063 query = parse_qs(parsed.query) 1064 query[key] = [value] # add or replace parameter 1065 1066 new_query = urlencode(query, doseq=True) 1067 new_url = parsed._replace(query=new_query) 1068 return str(urlunparse(new_url))
1078 def refresh_info_json( 1079 self, 1080 update_threshold: float, 1081 video_id, 1082 cookies=None, 1083 additional_options=None, 1084 include_dash=False, 1085 include_m3u8=False, 1086 ): 1087 # Check if time difference is greater than the threshold. If doesn't exist, subtraction of zero will always be true 1088 if time.time() - self.refresh_json.get("refresh_time", 0.0) > update_threshold: 1089 self.refresh_json, self.live_status = get_video_info( 1090 video_id=video_id, 1091 wait=False, 1092 cookies=cookies, 1093 additional_options=additional_options, 1094 include_dash=include_dash, 1095 include_m3u8=include_m3u8, 1096 clean_info_dict=True, 1097 ) 1098 1099 # Remove unnecessary items for info.json used purely for url refresh 1100 self.remove_format_segment_playlist_from_info_dict(self.refresh_json) 1101 1102 self.refresh_json.pop("thumbnails", None) 1103 self.refresh_json.pop("tags", None) 1104 self.refresh_json.pop("description", None) 1105 1106 # Add refresh time for reference 1107 self.refresh_json["refresh_time"] = time.time() 1108 1109 return self.refresh_json, self.live_status 1110 return self.refresh_json, self.live_status