ls_dlp

ls-dlp

PyPI Release Package CI

This is a forked project of livestream_dl.

A robust YouTube livestream downloader that combines the reliability of yt-dlp with the fragment-based downloading principles of ytarchive.

This tool focuses on using yt-dlp for stream information extraction, ensuring compatibility with YouTube's changing infrastructure. It is designed to handle live recording with the ability to recover streams if they become unavailable (e.g., go private) during the download process.

Key Features

  • Robust Downloading: Uses yt-dlp for extraction, minimizing breakage from YouTube updates.
  • Stream Recovery: capable of recovering streams that go private during recording (requires specific configuration).
  • Dual Write Modes: Supports both direct .ts file writing (low disk IO) and SQLite-backed downloading (safer for unstable connections).
  • Live Chat: Integrated support for downloading live chat (via yt-dlp) and bundling it into a ZIP file.
  • Channel Monitoring: Automated monitoring of channels for upcoming or live streams.
  • Protocol Fallbacks: Optional support for DASH and HLS (m3u8) protocols to avoid modifying yt-dlp source code.

Requirements

See Dockerfile

Installation

pip install ls-dlp

Usage

ls-dlp [OPTIONS] [VIDEO_URL_OR_ID]

Examples

Basic download:

ls-dlp \
  --threads 4 \
  --dash \
  --m3u8 \
  --write-thumbnail \
  --embed-thumbnail \
  --wait-for-video "60:600" \
  --clean-info-json \
  --remove-ip-from-json \
  --live-chat \
  --resolution "best" \
  --write-info-json \
  --log-level "INFO" -- "[VIDEO_ID]"

Download with live chat and embed thumbnail:

ls-dlp --live-chat --threads 4 --embed-thumbnail VIDEO_ID

Monitor a channel for streams:

ls-dlp --monitor-channel --threads 4 --dash --m3u8 --wait-for-video 60 CHANNEL_ID

Command Line Options

General Configuration

Option Default Description
ID None The video URL or ID (Positional argument).
--help, -h N/A Show the help message and exit.
--cookies None Path to a Netscape-formatted cookies file (required for age-gated or members-only content).
--temp-folder None Path for temporary files (database, segments). Supports yt-dlp output formatting.
--output %(fulltitle)s (%(id)s) Path/filename for output files. Supports yt-dlp output formatting. A good example is: "[%(release_date,upload_date)s] %(fulltitle).120B [%(channel)s] (%(id)s)"
--ext None Force the extension of the final video file (e.g., .mp4, .mkv).
--json-file None Path to an existing yt-dlp info.json file. Overrides ID and skips retrieving URLs from the web.
--write-ffmpeg-command False Writes the final FFmpeg command used to merge the file to a .txt file for debugging or manual merging.

Download & Formatting

Option Default Description
--resolution best Desired resolution. Can be best, audio_only, or a custom yt-dlp format filter.
-Audio is always set to "ba" (best audio) regardless of filters.
-"best" is converted to "bv".
-If unspecified, a prompt will appear.
--custom-sort None Custom sorting algorithm for formats based on yt-dlp sorting syntax.
--threads 1 Number of download threads per format. Total threads = this value × 2 (for video+audio).
--batch-size 5 Number of segments downloaded before the temporary database is committed to disk (reduces disk IO).
--segment-retries 10 Number of times to retry downloading a specific segment before failing.
--dash False Use DASH URLs as fallback. Does not require yt-dlp modification but prevents stream recovery if DASH URLs are used.
--m3u8 False Use HLS (m3u8) URLs as fallback. Combined audio/video streams (halves requests). Does not require yt-dlp modification but prevents recovery if m3u8 URLs are used.
--force-m3u8 False Forces the use of m3u8 stream URLs.
--wait-for-video None Wait time (int) or range (min:max) to wait for a video to start or become available. Refer to yt-dlp's documentation
--ytdlp-options None JSON string of additional yt-dlp options (overwrites conflicts). E.g '{"extractor_args":{"youtubepot-bgutilhttp":{"base_url":["http://bgutil-provider:4416"]}}}' to use Potoken retrieval. Refer to full options for more examples.

Processing & Output

Option Default Description
--no-merge False Do not merge video and audio using FFmpeg after download.
--merge False Force merging video using FFmpeg (overrides --no-merge).
--write-thumbnail False Save the video thumbnail to a separate file.
--embed-thumbnail False Embed the thumbnail into the final video file (ignored if --no-merge is active).
--write-info-json False Write the video metadata to an info.json file.
--write-description False Write the video description to a separate text file.
--keep-temp-files False Keep all temporary files (database and TS files) after finishing.
--keep-ts-files False Keep the intermediate TS files but delete the database.
--live-chat False Download live chat (requires yt-dlp).
--stop-chat-when-done 300 Max seconds to wait for chat download to finish after stream ends.

Database & Storage Modes

Option Default Description
--database-in-memory False Keep the segment database in RAM. High memory usage; not recommended for long streams.
--direct-to-ts False Write directly to a .ts file instead of using a SQLite database. See "Downloader methods" below.
--keep-database-file False Keep the database file after completion. If using --direct-to-ts, keeps the state file.

Recovery & Resilience

Option Default Description
--recovery False Puts the downloader directly into stream recovery mode.
--force-recover-merge False Forces merging to the final file even if not all segments were successfully recovered.
--recovery-failure-tolerance 0 Max number of fragments allowed to fail during recovery without throwing a final error.
--wait-limit 0 Max wait intervals (~10s each) for new segments. If 0, waits until status changes to post_live or was_live.

Network & Logging

Option Default Description
--proxy None Proxy URL (string) or JSON string for multiple methods. First proxy used for yt-dlp/chat.
--ipv4 False Force IPv4 only.
--ipv6 False Force IPv6 only.
--log-level INFO Logging level: DEBUG, VERBOSE, INFO, WARNING, ERROR, CRITICAL. Verbose logging is a custom level that includes the INFO logs of yt-dlp.
--no-console False Disable printing log messages to the console.
--log-file None Path to a file where log messages will be saved.
--stats-as-json False Prints download statistics as a JSON string (bypasses log level).
--new-line False Ensures console messages always print to a new line (useful for some terminals).

Channel Monitoring

Option Default Description
--monitor-channel False Enables channel monitoring mode. Use the Channel ID in the ID argument.
--members-only False Monitor the 'Members Only' playlist instead of public streams. Requires cookies.
--upcoming-lookahead 24 Maximum time (in hours) to look ahead for upcoming videos to schedule.
--playlist-items 50 Maximum number of playlist items to check when monitoring.

Metadata Privacy

Option Default Description
--remove-ip-from-json False Replaces IP addresses in info.json with 0.0.0.0.
--clean-urls False Removes potentially identifiable stream URLs from info.json.
--clean-info-json False Enables yt-dlp's internal clean-info-json option.

Full options

$ ls-dlp --help
usage: runner.py [-h] [--resolution RESOLUTION] [--custom-sort CUSTOM_SORT] [--threads THREADS] [--batch-size BATCH_SIZE] [--segment-retries SEGMENT_RETRIES] [--no-merge] [--merge] [--cookies COOKIES] [--output OUTPUT] [--ext EXT] [--temp-folder TEMP_FOLDER] [--write-thumbnail] [--embed-thumbnail]      
                 [--write-info-json] [--write-description] [--keep-temp-files] [--keep-ts-files] [--live-chat] [--keep-database-file] [--recovery] [--force-recover-merge] [--recovery-failure-tolerance RECOVERY_FAILURE_TOLERANCE] [--wait-limit WAIT_LIMIT] [--database-in-memory] [--direct-to-ts]
                 [--wait-for-video WAIT_FOR_VIDEO] [--json-file JSON_FILE] [--remove-ip-from-json] [--clean-urls] [--clean-info-json] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-console] [--log-file LOG_FILE] [--write-ffmpeg-command] [--stats-as-json] [--ytdlp-options YTDLP_OPTIONS] [--dash]
                 [--m3u8] [--force-m3u8] [--proxy [PROXY]] [--ipv4 | --ipv6] [--stop-chat-when-done STOP_CHAT_WHEN_DONE] [--new-line] [--monitor-channel] [--members-only] [--upcoming-lookahead UPCOMING_LOOKAHEAD] [--playlist-items PLAYLIST_ITEMS]
                 [ID]

Download YouTube livestream (https://github.com/CanOfSocks/livestream_dl)

positional arguments:
  ID                    The video URL or ID

options:
  -h, --help            show this help message and exit
  --resolution RESOLUTION
                        Desired resolution. Can be best, audio_only or a custom filter based off yt-dlp's format filtering: https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#filtering-formats. Audio will always be set as "ba" (best audio) regardless of filters set. "best" will be converted to "bv" A prompt     
                        will be displayed if no value is entered
  --custom-sort CUSTOM_SORT
                        Custom sorting algorithm for formats based off yt-dlp's format sorting: https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#sorting-formats
  --threads THREADS     Number of download threads per format. This will be 2x for an video and audio download. Default: 1
  --batch-size BATCH_SIZE
                        Number of segments before the temporary database is committed to disk. This is useful for reducing disk access instances. Default: 5
  --segment-retries SEGMENT_RETRIES
                        Number of times to retry grabbing a segment. Default: 10
  --no-merge            Don't merge video using ffmpeg
  --merge               Merge video using ffmpeg, overrides --no-merge
  --cookies COOKIES     Path to cookies file
  --output OUTPUT       Path/file name for output files. Supports yt-dlp output formatting
  --ext EXT             Force extension of video file. E.g. '.mp4'
  --temp-folder TEMP_FOLDER
                        Path for temporary files. Supports yt-dlp output formatting
  --write-thumbnail     Write thumbnail to file
  --embed-thumbnail     Embed thumbnail into final file. Ignored if --no-merge is used
  --write-info-json     Write info.json to file
  --write-description   Write description to file
  --keep-temp-files     Keep all temp files i.e. database and/or ts files
  --keep-ts-files       Keep all ts files
  --live-chat           Get Live chat
  --keep-database-file  Keep database file. If using with --direct-to-ts, this keeps the state file
  --recovery            Puts downloader into stream recovery mode
  --force-recover-merge
                        Forces merging to final file even if all segments could not be recovered
  --recovery-failure-tolerance RECOVERY_FAILURE_TOLERANCE
                        Maximum number of fragments that fail to download (exceed the retry limit) and not throw an error. May cause unexpected issues when merging to .ts file and remuxing. Default: 0
  --wait-limit WAIT_LIMIT
                        Set maximum number of wait intervals for new segments. Each wait interval is ~10s (e.g. a value of 20 would be 200s). A minimum of value of 20 is recommended. Stream URLs are refreshed every 10 intervals. A value of 0 wait until the video moves into 'was_live' or 'post_live' status.       
                        Default: 0
  --database-in-memory  Keep stream segments database in memory. Requires a lot of RAM (Not recommended)
  --direct-to-ts        Write directly to ts file instead of database. May use more RAM if a segment is slow to download. This overwrites most database options
  --wait-for-video WAIT_FOR_VIDEO
                        Wait time (int) or Minimum and maximum (min:max) interval to wait for a video
  --json-file JSON_FILE
                        Path to existing yt-dlp info.json file. Overrides ID and skips retrieving URLs
  --remove-ip-from-json
                        Replaces IP entries in info.json with 0.0.0.0
  --clean-urls          Removes stream URLs from info.json that contain potentially identifiable information. These URLs are usually useless once they have expired
  --clean-info-json     Enables yt-dlp's 'clean-info-json' option
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level. Default is INFO.
  --no-console          Do not log messages to the console.
  --log-file LOG_FILE   Path to the log file where messages will be saved.
  --write-ffmpeg-command
                        Writes FFmpeg command to a txt file
  --stats-as-json       Prints stats as a JSON formatted string. Bypasses logging and prints regardless of log level
  --ytdlp-options YTDLP_OPTIONS
                        Additional yt-dlp options as a JSON string. Overwrites any options that are already defined by other options. Available options: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py#L183. E.g. '{"extractor_args": {"youtube": {"player_client": ["web_creator"]}, "youtubepot-     
                        bgutilhttp":{ "base_url": ["http://10.1.1.40:4416"]}}}' if you have installed the potoken plugin
  --dash                Gets any available DASH urls as a fallback to adaptive URLs. Dash URLs do not require yt-dlp modification to be used, but can't be used for stream recovery and can cause large info.json files when a stream is in the 'post_live' status
  --m3u8                Gets any available m3u8 urls as a fallback to adaptive URLs. m3u8 URLs do not require yt-dlp modification to be used, but can't be used for stream recovery. m3u8 URLs provide both video and audio in each fragment and could allow for the amount of segment download requests to be halved     
  --force-m3u8          Forces use of m3u8 stream URLs
  --proxy [PROXY]       (Requires testing) Specify proxy to use for web requests. Can be a string for a single proxy or a JSON formatted string to specify multiple methods. For multiple, refer to format https://requests.readthedocs.io/en/latest/user/advanced/#proxies. The first proxy specified will be used for   
                        yt-dlp and live chat functions.
  --ipv4                Force IPv4 only
  --ipv6                Force IPv6 only
  --stop-chat-when-done STOP_CHAT_WHEN_DONE
                        Wait a maximum of X seconds after a stream is finished to download live chat. Default: 300. This is useful if waiting for chat to end causes hanging.
  --new-line            Console messages always print to new line. (Currently only ensured for stats output)
  --monitor-channel     Use monitor channel feature (Alpha). Specify channel ID in 'ID' argument
  --members-only        Monitor 'Members Only' playlist for streams instead of 'Streams' playlist. Requires cookies.
  --upcoming-lookahead UPCOMING_LOOKAHEAD
                        Maximum time (in hours) to start a downloader instance for a video. Default: 24
  --playlist-items PLAYLIST_ITEMS
                        Maximum number of playlist items to check. Default: 50

Downloader methods

There are two download methods available for livestream downloading, using a SQLite database and writing directly to a ts file. For stream recovery, only the SQLite option is available. Merging to a video or audio file (.mp4, .mkv, .ogg etc) will require an extra write of information. The default method is the SQLite method.

Direct writing to a .ts file

Writing directly to a ts file helps reduce the number of writes to a disk during the recording as it is only necessary to write to the final ts file. This may reduce disk wear over a long period of time of downloading many streams. As segments are downloaded, the sequence number of the segment is used to decide which segment will be appended to the ts file next. If the "next" segment is not available to be written to the disk, other segments will be stored in RAM until the respective previous segment has been written to the disk. This performs in a similar way to ytarchive. To reduce file opening and closing actions on the file system, if multiple sequential segments can be written at once, they will be written into the file at once. A state file is saved each time segments are written to the disk, saving the latest segment and size of the file, so the downloader doesn't need to re-download segments should the downloading session stop for any reason. Segment downloads are not guaranteed to finish in a sequential order if more than 1 thread is used. If a segment is particularly slow at downloading, this may increase ram usage significantly while subsequent segments are saved in RAM.

For regular livestream recording that doesn't experience frequent segment download errors, using the direct writing method will work well.

SQLite

The SQLite downloader method is used to improve handling of non-sequential successful segment downloads. This works by creating and using a basic table of the segment number as an ID and a blob to store the segment data. Once all segments are downloaded, a query is executed to sort all of the downloaded segments into the correct order and saved to a .ts file. This helps significantly to manage downloaded segments when failures occur often, such as unavailable stream recovery. By writing to the database and a final .ts file, this will require 2 full writes of the downloaded data, which may increase wear on flash-based systems. For most users, this increased wear will not be significant in the long-term. This improves over saving individual segment files like ytarchive-raw-go as all the downloaded segments are encapsulated into a single file, making it easier for the file system to handle. In the SQLite method, the existence of a downloaded segment is checked before it is downloaded. This allows failed segments to be "looped back to" later without causing other slowdowns, and ensures some information is saved for a segment (even if it is empty, as is the case sometimes).


Known Issues

  • Concurrent Futures: The downloader uses concurrent.futures for thread management. While robust, stopping the downloader gracefully (e.g., via Keyboard Interrupt) can sometimes be delayed if a thread is stuck on a request.
  • Stream Recovery: This feature is currently in a "semi-broken" state and may not work reliably for all stream types.
 1""".. include:: ../README.md"""
 2
 3from __future__ import annotations
 4
 5import importlib.metadata
 6
 7from .download_live import LiveStreamDownloader
 8
 9try:
10    __version__ = importlib.metadata.version(__name__)
11except importlib.metadata.PackageNotFoundError:
12    __version__ = "0.0.0"
13
14
15__all__ = (
16    "LiveStreamDownloader",
17    "__version__",
18)
class LiveStreamDownloader:
  39class LiveStreamDownloader:
  40    def __init__(
  41        self,
  42        kill_all: threading.Event = threading.Event(),
  43        cleanup: threading.Event = threading.Event(),
  44        logger: logging.Logger | logging.LoggerAdapter | None = None,
  45        kill_this: threading.Event | None = None,
  46    ) -> None:
  47        if logger:
  48            self.logger = logger
  49        else:
  50            # 1. Create a Named Logger instance and assign it to self.logger
  51            self.logger = logging.getLogger(self.__class__.__name__)
  52            # Ensure it processes messages (important if you don't call setup_logging immediately)
  53        if isinstance(self.logger, logging.Logger):
  54            self.logger.propagate = False
  55        self.kill_this = kill_this or threading.Event()
  56
  57        # Global state converted to instance attributes
  58        self.kill_all = kill_all
  59        self.cleanup = cleanup
  60        self.live_chat_result = None
  61        self.chat_timeout = None
  62        # File name dictionary
  63        self.file_names = {"databases": [], "streams": {}}
  64        self.stats = {}
  65
  66        self.refresh_json = {}
  67        self.live_status = ""
  68        self.lock: threading.Lock = threading.Lock()
  69
  70    # Create runner function for each download format
  71    def download_stream(
  72        self,
  73        info_dict,
  74        resolution,
  75        batch_size=5,
  76        max_workers=1,
  77        folder=None,
  78        file_name=None,
  79        keep_database=False,
  80        cookies=None,
  81        retries=5,
  82        yt_dlp_options=None,
  83        proxies=None,
  84        yt_dlp_sort=None,
  85        include_dash=False,
  86        include_m3u8=False,
  87        force_m3u8=False,
  88        manifest=0,
  89        **kwargs,
  90    ):
  91        try:
  92            download_params = locals().copy()
  93            download_params.update({"download_function": self.download_stream})
  94            file = None
  95            filetype = None
  96
  97            with DownloadStream(
  98                info_dict,
  99                resolution=resolution,
 100                batch_size=batch_size,
 101                max_workers=max_workers,
 102                folder=folder,
 103                file_name=file_name,
 104                cookies=cookies,
 105                fragment_retries=retries,
 106                yt_dlp_options=yt_dlp_options,
 107                proxies=proxies,
 108                yt_dlp_sort=yt_dlp_sort,
 109                include_dash=include_dash,
 110                include_m3u8=include_m3u8,
 111                force_m3u8=force_m3u8,
 112                download_params=download_params,
 113                livestream_coordinator=self,
 114                **kwargs,
 115            ) as downloader:
 116                self.stats["status"] = "Recording"
 117                downloader.live_dl()
 118                file_name = downloader.combine_segments_to_file(downloader.merged_file_name)
 119                if not keep_database:
 120                    self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}")
 121                    downloader.delete_temp_database()
 122                elif downloader.temp_db_file != ":memory:":
 123                    database_file = FileInfo(downloader.temp_db_file, file_type="database", format=downloader.format)
 124                    self.file_names["databases"].append(database_file)
 125
 126                file = FileInfo(file_name, file_type=downloader.type, format=downloader.format)
 127                filetype = downloader.type
 128
 129            self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file})
 130
 131            return file, filetype
 132        except Exception:
 133            self.logger.exception("Unexpected error occurred while downloading stream")
 134            raise
 135
 136    # Create runner function for each download format
 137    def download_stream_direct(
 138        self,
 139        info_dict,
 140        resolution,
 141        batch_size,
 142        max_workers,
 143        folder=None,
 144        file_name=None,
 145        keep_state=False,
 146        cookies=None,
 147        retries=5,
 148        yt_dlp_options=None,
 149        proxies=None,
 150        yt_dlp_sort=None,
 151        include_dash=False,
 152        include_m3u8=False,
 153        force_m3u8=False,
 154        manifest=0,
 155        **kwargs,
 156    ):
 157        try:
 158            download_params = locals().copy()
 159            download_params.update({"download_function": self.download_stream_direct})
 160            file = None
 161            filetype = None
 162
 163            with DownloadStreamDirect(
 164                info_dict,
 165                resolution=resolution,
 166                max_workers=max_workers,
 167                folder=folder,
 168                file_name=file_name,
 169                cookies=cookies,
 170                fragment_retries=retries,
 171                yt_dlp_options=yt_dlp_options,
 172                proxies=proxies,
 173                yt_dlp_sort=yt_dlp_sort,
 174                include_dash=include_dash,
 175                include_m3u8=include_m3u8,
 176                force_m3u8=force_m3u8,
 177                download_params=download_params,
 178                livestream_coordinator=self,
 179                **kwargs,
 180            ) as downloader:
 181                self.stats["status"] = "Recording"
 182                file_name = downloader.live_dl()
 183                file = FileInfo(file_name, file_type=downloader.type, format=downloader.format)
 184                filetype = downloader.type
 185                downloader.delete_state_file()
 186
 187            self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file})
 188            return file, filetype
 189        except Exception:
 190            self.logger.exception("Unexpected error occurred while downloading stream")
 191            raise
 192
 193    def recover_stream(
 194        self,
 195        info_dict,
 196        resolution,
 197        batch_size=5,
 198        max_workers=5,
 199        folder=None,
 200        file_name=None,
 201        keep_database=False,
 202        cookies=None,
 203        retries=5,
 204        yt_dlp_options=None,
 205        proxies=None,
 206        yt_dlp_sort=None,
 207        force_merge=False,
 208        recovery_failure_tolerance=0,
 209        manifest=0,
 210        stream_urls: list | None = None,
 211        no_merge=False,
 212        **kwargs,
 213    ):
 214        if stream_urls is None:
 215            stream_urls = []
 216        try:
 217            file = None
 218            filetype = None
 219
 220            with StreamRecovery(
 221                info_dict,
 222                resolution=resolution,
 223                batch_size=batch_size,
 224                max_workers=max_workers,
 225                folder=folder,
 226                file_name=file_name,
 227                cookies=cookies,
 228                fragment_retries=retries,
 229                proxies=proxies,
 230                yt_dlp_sort=yt_dlp_sort,
 231                livestream_coordinator=self,
 232                stream_urls=stream_urls,
 233                **kwargs,
 234            ) as downloader:
 235                self.stats["status"] = "Recording"
 236                result = downloader.live_dl()
 237                if (force_merge or result <= 0 or result <= recovery_failure_tolerance) and (not no_merge):
 238                    if result > 0:
 239                        self.logger.warning(
 240                            f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting"
 241                        )
 242                    file_name = downloader.combine_segments_to_file(downloader.merged_file_name)
 243                    if not keep_database:
 244                        self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}")
 245                        downloader.delete_temp_database()
 246                    elif downloader.temp_db_file != ":memory:":
 247                        database_file = FileInfo(
 248                            downloader.temp_db_file, file_type="database", format=downloader.format
 249                        )
 250                        self.file_names["databases"].append(database_file)
 251                else:
 252                    msg = f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting"
 253                    raise VideoDownloadError(msg)
 254                file = FileInfo(file_name, file_type=downloader.type, format=downloader.format)
 255                filetype = downloader.type
 256
 257            self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file})
 258        except Exception:
 259            self.logger.exception("Unexpected error occurred while downloading stream")
 260            raise
 261        return file, filetype
 262
 263    def submit_download(
 264        self, executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False
 265    ):
 266        extra_kwargs = {}
 267
 268        # Options only not used by recovery
 269        if not options.get("recovery", False):
 270            extra_kwargs.update(
 271                {
 272                    "include_dash": options.get("dash", False),
 273                    "include_m3u8": options.get("m3u8", False),
 274                    "force_m3u8": options.get("force_m3u8", False),
 275                }
 276            )
 277
 278        # Select the function
 279        if options.get("recovery", False):
 280            func = self.recover_stream
 281            extra_kwargs.update(
 282                {
 283                    "force_merge": options.get("force_recovery_merge", False),
 284                    "recovery_failure_tolerance": options.get("recovery_failure_tolerance", 0),
 285                }
 286            )
 287        elif options.get("direct_to_ts", False):
 288            func = self.download_stream_direct
 289            extra_kwargs.update(
 290                {
 291                    "keep_state": options.get("keep_temp_files", False) or options.get("keep_database_file", False),
 292                }
 293            )
 294        else:
 295            func = self.download_stream
 296            extra_kwargs.update(
 297                {
 298                    "keep_database": options.get("keep_temp_files", False) or options.get("keep_database_file", False),
 299                }
 300            )
 301
 302        kwargs = {
 303            "info_dict": info_dict,
 304            "resolution": "audio_only" if is_audio else resolution,
 305            "batch_size": options.get("batch_size", 1),
 306            "max_workers": options.get("threads", 1),
 307            "folder": download_folder,
 308            "file_name": file_name,
 309            "retries": options.get("segment_retries"),
 310            "cookies": options.get("cookies"),
 311            "yt_dlp_options": options.get("ytdlp_options", None),
 312            "proxies": options.get("proxy", None),
 313            "yt_dlp_sort": options.get("custom_sort", None),
 314        }
 315
 316        if extra_kwargs:
 317            kwargs.update(extra_kwargs)
 318
 319        # Add any extra values not yet existing
 320        for key, value in options.items():
 321            kwargs.setdefault(key, value)
 322
 323        self.logger.debug(f"Starting executor with: {json.dumps(kwargs)}")
 324
 325        # Submit to executor
 326        future = executor.submit(func, **kwargs)
 327        futures.add(future)
 328        return future
 329
 330    # Multithreaded function to download new segments with delayed commit after a batch
 331    def download_segments(
 332        self, info_dict, resolution="best", options=None, thread_event: threading.Event | None = None
 333    ) -> None:
 334        if options is None:
 335            options = {}
 336        futures = set()
 337
 338        if thread_event is not None:
 339            self.kill_all = thread_event
 340
 341        self.stats["id"] = info_dict.get("id", None)
 342        self.stats["status"] = "Starting"
 343        self.logger.debug(json.dumps(options, indent=4))
 344        outputFile = self.output_filename(info_dict=info_dict, outtmpl=options.get("output"))
 345        file_name = None
 346        # Requires testing
 347        if options.get("temp_folder") is not None and options.get("temp_folder") != os.path.dirname(outputFile):
 348            _output_folder, file_name = os.path.split(outputFile)
 349            download_folder = self.output_filename(info_dict=info_dict, outtmpl=options.get("temp_folder"))
 350            options["temp_folder"] = download_folder
 351        else:
 352            download_folder, file_name = os.path.split(outputFile)
 353        options["filename"] = file_name
 354        if download_folder:
 355            os.makedirs(download_folder, exist_ok=True)
 356
 357        with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
 358            done = not_done = set()
 359            try:
 360                # Download auxiliary files (thumbnail, info,json etc)
 361                auxiliary_thread = executor.submit(self.download_auxiliary_files, info_dict=info_dict, options=options)
 362                futures.add(auxiliary_thread)
 363
 364                live_chat_thread = None
 365                if options.get("live_chat", False) is True:
 366                    live_chat_thread = threading.Thread(
 367                        target=self.download_live_chat, args=(info_dict, options), daemon=True
 368                    )
 369                    live_chat_thread.start()
 370
 371                format_check = Formats().getFormatURL(
 372                    info_json=info_dict,
 373                    resolution=resolution,
 374                    sort=options.get("custom_sort"),
 375                    include_dash=(options.get("dash", False) and not options.get("recovery", False)),
 376                    include_m3u8=options.get("m3u8", False),
 377                    force_m3u8=options.get("force_m3u8", False),
 378                )
 379                if not format_check:
 380                    msg = "Resolution is not valid or does not exist in stream"
 381                    raise ValueError(msg)
 382                # For use of specificed format. Expects two values, but can work with more
 383                # Video + Audio
 384
 385                if resolution.lower() not in ("audio_only", "ba"):
 386                    # Video
 387                    self.submit_download(
 388                        executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False
 389                    )
 390                    if format_check.protocol != "m3u8_native":
 391                        # Audio
 392                        self.submit_download(
 393                            executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True
 394                        )
 395                # Audio-only
 396                else:
 397                    self.submit_download(
 398                        executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True
 399                    )
 400
 401                while True:
 402                    if self.kill_all.is_set() or self.kill_this.is_set():
 403                        msg = "Thread kill event is set, ending..."
 404                        raise KeyboardInterrupt(msg)
 405                    done, not_done = concurrent.futures.wait(
 406                        futures, timeout=1, return_when=concurrent.futures.FIRST_COMPLETED
 407                    )
 408                    # Continuously check for completion or interruption
 409                    for future in done:
 410                        result, type = future.result()
 411                        self.logger.info(f"\033[31m{result}\033[0m")
 412
 413                        if type == "auxiliary":
 414                            self.file_names.update(result)
 415
 416                            # Remove parts that would not be needed by stream downloaders
 417                            self.remove_format_segment_playlist_from_info_dict(info_dict=info_dict)
 418                            info_dict.pop("thumbnails", None)
 419                            # Remove no longer required parts for the rest of the downloader
 420                            # info_dict = self.trim_info_json(info_dict=info_dict, keys_to_keep={"id", "ext", "upload_date", "original_url", "description", "fulltitle", "channel"})
 421
 422                        elif str(type).lower() == "video":
 423                            # self.file_names['video'] = result
 424                            pass
 425                        elif str(type).lower() == "audio":
 426                            # self.file_names['audio'] = result
 427                            pass
 428                        elif str(type).lower() == "live_chat":
 429                            pass
 430                        else:
 431                            self.file_names[str(type)] = result
 432
 433                        futures.discard(future)
 434
 435                    if len(not_done) <= 0:
 436                        break
 437                    # else:
 438                    #    time.sleep(0.9)
 439                    self.print_stats(options=options)
 440
 441                if live_chat_thread and live_chat_thread.is_alive():
 442                    self.logger.info("Media downloads finished, waiting for live chat to end...")
 443                    if options.get("stop_chat_when_done"):
 444                        self.chat_timeout = time.time()
 445                    while live_chat_thread.is_alive() and not (self.kill_all.is_set() or self.kill_this.is_set()):
 446                        time.sleep(0.1)
 447
 448            except KeyboardInterrupt:
 449                self.kill_this.set()
 450                self.logger.debug("Keyboard interrupt detected")
 451                if len(not_done) > 0:
 452                    self.logger.debug("Cancelling remaining threads")
 453                for future in not_done:
 454                    _ = future.cancel()
 455                done, not_done = concurrent.futures.wait(
 456                    futures, timeout=5, return_when=concurrent.futures.ALL_COMPLETED
 457                )
 458                executor.shutdown(wait=False, cancel_futures=True)
 459                self.logger.debug("Shutdown all threads")
 460                self.stats["status"] = "Cancelled"
 461                raise
 462
 463        self.stats["status"] = "Muxing"
 464        self.create_mp4(file_names=self.file_names, info_dict=info_dict, options=options)
 465
 466        self.stats["status"] = "Moving"
 467        self.move_to_final(options=options, output_file=outputFile, file_names=self.file_names)
 468        self.stats["status"] = "Finished"
 469        # move_to_final(info_dict, options, file_names)
 470
 471    def output_filename(self, info_dict, outtmpl):
 472        return str(YoutubeDL().prepare_filename(info_dict, outtmpl=outtmpl)).replace(
 473            "%", "%"
 474        )  # Replace normal percent sign with unicode version to prevent ffmpeg errors
 475
 476    def move_to_final(self, options, output_file, file_names) -> None:
 477        def maybe_move(key, dest_func, file_names_dict=file_names, option_flag=None) -> None:
 478            """key: key in file_names
 479            dest_func: func -> pathlib -> string dest path
 480            delete_if_false: if True, delete when option_flag not set
 481            option_flag: name of boolean option (write_thumbnail etc)
 482            """
 483            f = file_names_dict.get(key)
 484            if not f:
 485                return
 486            try:
 487                # deletion case (thumbnail / ffmpeg)
 488                if option_flag is not None and not options.get(option_flag, False):
 489                    self.logger.info(f"Removing {f.absolute()}")
 490                    f.unlink(missing_ok=True)
 491                    file_names_dict.pop(key, None)
 492                    return
 493
 494                dest = dest_func(f)
 495                if str(f).strip() != str(dest).strip():
 496                    self.logger.info(f"Moving {f.absolute()} → {dest}")
 497                    shutil.move(f.absolute(), dest)
 498                else:
 499                    self.logger.debug(f"{f.absolute()} is already in final destination")
 500            except Exception as e:
 501                self.logger.exception(f"unable to move {key}: {e}")
 502
 503        # ensure output dir exists
 504        out_dir = os.path.dirname(output_file)
 505        if out_dir:
 506            os.makedirs(out_dir, exist_ok=True)
 507
 508        # === individual file handlers ===
 509
 510        maybe_move("thumbnail", lambda f: f"{output_file}{f.suffix}", option_flag="write_thumbnail")
 511
 512        maybe_move(
 513            "info_json",
 514            lambda f: f"{output_file}.info.json",
 515        )
 516
 517        maybe_move(
 518            "description",
 519            lambda f: f"{output_file}{f.suffix}",
 520        )
 521
 522        stream_manifests = list(self.file_names["streams"].items())
 523        for manifest, stream in stream_manifests:
 524            stream_output_file = output_file
 525            if len(stream_manifests) > 1:
 526                stream_output_file = f"{output_file}.{manifest}"
 527            maybe_move("video", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream)
 528
 529            maybe_move("audio", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream)
 530
 531            maybe_move("merged", lambda f: f"{stream_output_file}{f.suffix}", file_names_dict=stream)
 532
 533        maybe_move(
 534            "live_chat",
 535            lambda f: f"{output_file}.live_chat.zip",
 536        )
 537
 538        # special: databases = list
 539        try:
 540            for f in file_names.get("databases", []):
 541                dest = f"{output_file}.{f._format}{f.suffix}"
 542                if str(f).strip() != str(dest).strip():
 543                    self.logger.info(f"Moving {f.absolute()} → {dest}")
 544                    shutil.move(f.absolute(), dest)
 545        except Exception as e:
 546            self.logger.exception(f"unable to move database files: {e}")
 547
 548        # special: ffmpeg_cmd
 549        maybe_move("ffmpeg_cmd", lambda f: f"{output_file}.ffmpeg.txt", option_flag="write_ffmpeg_command")
 550
 551        # remove temp folder
 552        """
 553        if options.get('temp_folder', None) is not None:
 554            try:
 555                os.rmdir(options.get('temp_folder'))
 556            except OSError as e:
 557                if e.errno == errno.ENOTEMPTY:
 558                    self.logger.warning(f"Error: Directory not empty: {e.filename}")
 559                else:
 560                    self.logger.exception(f"Error removing temp folder: {e}")
 561            except Exception as e:
 562                self.logger.exception(f"Error removing temp folder: {e}")
 563        """
 564        self.logger.info("Finished moving files from temporary directory to output destination")
 565
 566    def download_live_chat(self, info_dict, options):
 567        filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id")
 568
 569        base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename
 570
 571        class YTDLP_Chat_logger(YTDLPLogger):
 572            def __init__(self, logger: logging.Logger | logging.LoggerAdapter | None = None) -> None:
 573                if logger is None:
 574                    logger = logging.getLogger()
 575                super().__init__(logger=logger)
 576
 577            def prefix(self, msg: str):
 578                if msg.startswith("[live-chat] "):
 579                    return msg
 580                return f"[live-chat] {msg}"
 581
 582            def debug(self, msg) -> None:
 583                super().debug(self.prefix(msg))
 584
 585            def info(self, msg) -> None:
 586                super().info(self.prefix(msg))
 587
 588            def warning(self, msg) -> None:
 589                super().warning(self.prefix(msg))
 590
 591            def error(self, msg) -> None:
 592                super().error(self.prefix(msg))
 593
 594        logger = self.logger
 595
 596        ydl_opts = {
 597            "skip_download": True,  # Skip downloading video/audio
 598            "logger": YTDLP_Chat_logger(logger=logger),
 599            #'quiet': True,
 600            "cookiefile": options.get("cookies", None),
 601            "retries": 25,
 602            "concurrent_fragment_downloads": 3,
 603            #'live_from_start': True,
 604            "writesubtitles": True,  # Extract subtitles (live chat)
 605            "subtitlesformat": "json",  # Set format to JSON
 606            "subtitleslangs": ["live_chat"],  # Only extract live chat subtitles
 607            "outtmpl": base_output,  # Save to a JSON file
 608        }
 609        self.logger.debug(options.get("ytdlp_options", {}))
 610        ydl_opts.update(options.get("ytdlp_options", {}))
 611
 612        livechat_filename = base_output + ".live_chat.json"
 613
 614        self.logger.info(f"Downloading live chat to: {livechat_filename}")
 615
 616        # Download live chat using yt-dlp
 617        with YoutubeDL(ydl_opts) as ydl:
 618            try:
 619                ydl.download([info_dict.get("original_url", info_dict.get("webpage_url", info_dict.get("url")))])
 620            except Exception as e:
 621                self.logger.exception(f"Failed to download live chat: {e}")
 622
 623        time.sleep(1)
 624        part_file = f"{livechat_filename}.part"
 625        if os.path.exists(part_file):
 626            # Append part to
 627            chunk_size = 1024 * 1024 * 10  # number of characters per chunk, up to 10M characters
 628            with open(part_file, encoding="utf-8") as fa, open(livechat_filename, "a", encoding="utf-8") as fb:
 629                while chunk := fa.read(chunk_size):
 630                    fb.write(chunk)
 631            os.remove(f"{livechat_filename}.part")
 632
 633        try:
 634            if not os.path.exists(livechat_filename):
 635                self.logger.warning(f"Live chat file not found: {livechat_filename}")
 636                return None, None
 637
 638            zip_filename = base_output + ".live_chat.zip"
 639            with zipfile.ZipFile(
 640                zip_filename, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=9, allowZip64=True
 641            ) as zipf:
 642                zipf.write(livechat_filename, arcname=os.path.basename(livechat_filename))
 643            os.remove(livechat_filename)
 644            live_chat = {"live_chat": FileInfo(zip_filename, file_type="live_chat")}
 645            self.file_names.update(live_chat)
 646            return live_chat, "live_chat"
 647        except Exception as e:
 648            self.logger.exception(f"\033[31m{e}\033[0m")
 649
 650    def replace_ip_in_json(self, file_name) -> None:
 651        pattern = re.compile(r"((?:[0-9]{1,3}\.){3}[0-9]{1,3})|((?:[a-f0-9]{1,4}:){7}[a-f0-9]{1,4})")
 652
 653        with open(file_name, encoding="utf-8") as file:
 654            content = file.read()
 655
 656        modified_content = re.sub(pattern, "0.0.0.0", content)
 657
 658        with open(file_name, "w", encoding="utf-8") as file:
 659            file.write(modified_content)
 660
 661    def remove_urls_from_json(self, file_name) -> None:
 662        with open(file_name, encoding="utf-8") as file:
 663            data = json.load(file)
 664
 665        if data.get("formats", None) is not None:
 666            for format in data["formats"]:
 667                if format.get("url") is not None:
 668                    format["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
 669
 670                if format.get("manifest_url") is not None:
 671                    format["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
 672
 673                format.pop("fragment_base_url", None)
 674                format.pop("fragments", None)
 675
 676        if data.get("thumbnails", None) is not None:
 677            for thumbnail in data["thumbnails"]:
 678                if thumbnail.get("url", None) is not None:
 679                    parsed_url = urlparse(thumbnail.get("url", ""))
 680                    thumbnail["url"] = f"{parsed_url.scheme}://{parsed_url.netloc}{parsed_url.path}"
 681
 682        if data.get("url", None) is not None:
 683            data["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
 684
 685        if data.get("manifest_url", None) is not None:
 686            data["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
 687
 688        data["removed_urls"] = True
 689
 690        with open(file_name, "w", encoding="utf-8") as file:
 691            json.dump(data, file)
 692
 693    def download_auxiliary_files(self, info_dict, options):
 694        filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id")
 695
 696        base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename
 697
 698        created_files = {}
 699
 700        logger = self.logger
 701
 702        class YTDLP_Auxiliary_logger(YTDLPLogger):
 703            def __init__(self, logger: logging.Logger | None = None) -> None:
 704                if logger is None:
 705                    logger = logging.getLogger()
 706                super().__init__(logger=logger)
 707
 708            def prefix(self, msg: str):
 709                if msg.startswith("[live-chat] "):
 710                    return msg
 711                return f"[live-chat] {msg}"
 712
 713            def debug(self, msg) -> None:
 714                super().debug(self.prefix(msg))
 715
 716            def info(self, msg) -> None:
 717                super().info(self.prefix(msg))
 718
 719            def warning(self, msg) -> None:
 720                super().warning(self.prefix(msg))
 721
 722            def error(self, msg) -> None:
 723                super().error(self.prefix(msg))
 724
 725        ydl_opts = {
 726            "skip_download": True,
 727            "quiet": True,
 728            #        'cookiefile': options.get('cookies', None),
 729            "writeinfojson": options.get("write_info_json", False),
 730            "writedescription": options.get("write_description", False),
 731            "writethumbnail": (options.get("write_thumbnail", False) or options.get("embed_thumbnail", False)),
 732            "outtmpl": base_output,
 733            "retries": 10,
 734            "logger": YTDLPLogger(logger=logger),
 735        }
 736        if options.get("proxy", None) is not None:
 737            ydl_opts["proxy"] = next(iter((options.get("proxy", None) or {}).values()), None)
 738
 739        with YoutubeDL(ydl_opts) as ydl:
 740            # base_name = ydl.prepare_filename(info_dict)
 741            # result = ydl.download_with_info_file(info_dict)
 742
 743            if ydl._write_info_json("video", info_dict, ydl.prepare_filename(info_dict, "infojson")) or os.path.exists(
 744                ydl.prepare_filename(info_dict, "infojson")
 745            ):
 746                created_files["info_json"] = FileInfo(
 747                    ydl.prepare_filename(info_dict, "infojson"), file_type="info_json"
 748                )
 749                try:
 750                    if options.get("remove_ip_from_json"):
 751                        self.replace_ip_in_json(created_files["info_json"].absolute())
 752                    if options.get("clean_urls"):
 753                        self.remove_urls_from_json(created_files["info_json"].absolute())
 754                except Exception as e:
 755                    self.logger.exception(str(e))
 756
 757            if ydl._write_description(
 758                "video", info_dict, ydl.prepare_filename(info_dict, "description")
 759            ) or os.path.exists(ydl.prepare_filename(info_dict, "description")):
 760                created_files["description"] = FileInfo(
 761                    ydl.prepare_filename(info_dict, "description"), file_type="description"
 762                )
 763
 764            thumbnails = ydl._write_thumbnails("video", info_dict, ydl.prepare_filename(info_dict, "thumbnail"))
 765
 766            if thumbnails:
 767                created_files["thumbnail"] = FileInfo(thumbnails[0][0], file_type="thumbnail")
 768
 769        return created_files, "auxiliary"
 770
 771    def create_mp4(self, file_names, info_dict, options):
 772        # self.logger.debug("Files: {0}".format(json.dumps(file_names)))
 773        stream_manifests = list(self.file_names["streams"].items())
 774        for manifest, stream in stream_manifests:
 775            index = 0
 776            thumbnail = None
 777            video = None
 778            ext = options.get("ext", None)
 779
 780            ffmpeg_builder = ["ffmpeg", "-y", "-hide_banner", "-nostdin", "-loglevel", "error", "-stats"]
 781
 782            # Add input files
 783            if stream.get("video", None):
 784                input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("video").absolute())]
 785                ffmpeg_builder.extend(input)
 786                video = index
 787                index += 1
 788
 789            if stream.get("audio", None):
 790                input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("audio").absolute())]
 791                ffmpeg_builder.extend(input)
 792                index += 1
 793                if video is None and ext is None:
 794                    ext = ".mka"
 795
 796            # Determine output path
 797            filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id")
 798
 799            base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename
 800
 801            if len(stream_manifests) > 1:
 802                base_output = f"{base_output}.{manifest}"
 803
 804            if ext is None:
 805                self.logger.debug("No extension detected, switching to yt-dlp decided video (defaulting to MP4)")
 806                ext = info_dict.get("ext", ".mp4")
 807            if ext is not None and not str(ext).startswith("."):
 808                ext = "." + str(ext)
 809            if not base_output.endswith(ext):
 810                base_output = base_output + ext
 811
 812            if file_names.get("thumbnail", None) and options.get("embed_thumbnail", True):
 813                if file_names.get("thumbnail").exists():
 814                    # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version
 815                    guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type)
 816
 817                    mime_type, _ = guess(file_names.get("thumbnail"))
 818
 819                    # If not jpeg or png, convert to png
 820                    if str(mime_type) not in ("image/jpeg", "image/png"):
 821                        self.logger.info(
 822                            "{} is not a JPG or PNG file, converting to png".format(file_names.get("thumbnail").name)
 823                        )
 824                        png_thumbnail = file_names.get("thumbnail").with_suffix(".png")
 825
 826                        thumbnail_conversion = [
 827                            "ffmpeg",
 828                            "-y",
 829                            "-hide_banner",
 830                            "-nostdin",
 831                            "-loglevel",
 832                            "error",
 833                            "-stats",
 834                            "-i",
 835                            str(file_names.get("thumbnail").absolute()),
 836                            "-c",
 837                            "png",
 838                            "-compression_level",
 839                            "9",
 840                            "-pred",
 841                            "mixed",
 842                            str(png_thumbnail.absolute()),
 843                        ]
 844                        try:
 845                            result = subprocess.run(
 846                                thumbnail_conversion,
 847                                capture_output=True,
 848                                text=True,
 849                                encoding="utf-8",
 850                                check=True,
 851                            )
 852                            self.logger.debug("Replacing thumbnail with .png version")
 853                            file_names.pop("thumbnail").unlink(missing_ok=True)
 854                            file_names["thumbnail"] = FileInfo(png_thumbnail, file_type="thumbnail")
 855
 856                        except subprocess.CalledProcessError as e:
 857                            self.logger.exception(e.stderr)
 858                            self.logger.critical(e)
 859
 860                    thumbnail = index
 861                    if ext.lower() != ".mkv":  # Don't add input file for mkv, use attach later
 862                        input = [
 863                            "-thread_queue_size",
 864                            "1024",
 865                            "-seekable",
 866                            "0",
 867                            "-i",
 868                            str(file_names.get("thumbnail").absolute()),
 869                        ]
 870                        ffmpeg_builder.extend(input)
 871                        index += 1
 872                else:
 873                    self.logger.error(
 874                        "Thumbnail file: {} is missing, continuing without embedding".format(
 875                            file_names.get("thumbnail").absolute()
 876                        )
 877                    )
 878
 879            # Add faststart
 880            ffmpeg_builder.extend(["-movflags", "faststart"])
 881
 882            # Add mappings
 883            for i in range(index):
 884                input = ["-map", str(i)]
 885                ffmpeg_builder.extend(input)
 886
 887            if thumbnail is not None:
 888                if ext.lower() == ".mkv":  # If file will be mkv, attach file instead
 889                    # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version
 890                    guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type)
 891                    mime_type, _ = guess(file_names.get("thumbnail"))
 892                    ffmpeg_builder.extend(
 893                        [
 894                            "-attach",
 895                            str(file_names.get("thumbnail").absolute()),
 896                            "-metadata:s:t:0",
 897                            "filename=cover{}".format(file_names.get("thumbnail").suffix),
 898                            "-metadata:s:t:0",
 899                            "mimetype={}".format(mime_type or "application/octet-stream"),
 900                        ]
 901                    )
 902
 903                else:  # For other formats, attach using disposition instead
 904                    ffmpeg_builder.extend([f"-disposition:{thumbnail}", "attached_pic"])
 905
 906            # Add Copy codec
 907            ffmpeg_builder.extend(["-c", "copy"])
 908
 909            # Add metadata
 910            ffmpeg_builder.extend(["-metadata", "DATE={}".format(info_dict.get("upload_date"))])
 911            ffmpeg_builder.extend(
 912                [
 913                    "-metadata",
 914                    "COMMENT={}\n{}".format(info_dict.get("original_url"), info_dict.get("description", "")),
 915                ]
 916            )
 917            ffmpeg_builder.extend(["-metadata", "TITLE={}".format(info_dict.get("fulltitle"))])
 918            ffmpeg_builder.extend(["-metadata", "ARTIST={}".format(info_dict.get("channel"))])
 919
 920            # Add output file to ffmpeg command
 921            ffmpeg_builder.append(str(Path(base_output).absolute()))
 922
 923            if options.get("write_ffmpeg_command", True):
 924                ffmpeg_command_file = f"{filename}.ffmpeg.txt"
 925                file_names["streams"][manifest]["ffmpeg_cmd"] = FileInfo(
 926                    self.write_ffmpeg_command(ffmpeg_builder, ffmpeg_command_file), file_type="ffmpeg_command"
 927                )
 928
 929            if not (options.get("merge", True)):
 930                return file_names
 931
 932            self.logger.debug("FFmpeg command: {}".format(" ".join(ffmpeg_builder)))
 933            self.logger.info(f"Executing ffmpeg. Outputting to {ffmpeg_builder[-1]}")
 934            try:
 935                result = subprocess.run(
 936                    ffmpeg_builder,
 937                    capture_output=True,
 938                    text=True,
 939                    encoding="utf-8",
 940                    check=True,
 941                )
 942                self.logger.debug(f"FFmpeg STDOUT: {result.stdout}")
 943                self.logger.debug(f"FFmpeg STDERR: {result.stderr}")
 944            except subprocess.CalledProcessError as e:
 945                self.logger.exception(e.stderr)
 946                self.logger.critical(e)
 947                raise
 948
 949            file_names["streams"][manifest]["merged"] = FileInfo(base_output, file_type="merged")
 950            self.logger.info(
 951                "Successfully merged files into: {}".format(file_names["streams"][manifest].get("merged").absolute())
 952            )
 953
 954            # Remove temp video and audio files
 955            if not (options.get("keep_ts_files") or options.get("keep_temp_files")):
 956                if file_names["streams"][manifest].get("video"):
 957                    self.logger.info("Removing {}".format(file_names["streams"][manifest].get("video").absolute()))
 958                    file_names["streams"][manifest].get("video").unlink(missing_ok=True)
 959                    file_names["streams"][manifest].pop("video", None)
 960                if file_names["streams"][manifest].get("audio"):
 961                    self.logger.info("Removing {}".format(file_names["streams"][manifest].get("audio").absolute()))
 962                    file_names["streams"][manifest].get("audio").unlink(missing_ok=True)
 963                    file_names["streams"][manifest].pop("audio", None)
 964
 965        return file_names
 966        # for file in file_names:
 967        #    os.remove(file)
 968
 969    def write_ffmpeg_command(self, command_array, filename):
 970        # Determine the platform
 971        """
 972        Builds a platform-compatible FFmpeg command with proper quoting.
 973
 974        Args:
 975            command_array (list): List of arguments to append to the command.
 976            filename: Filename to write command to
 977
 978        Returns:
 979            str: A properly quoted FFmpeg command.
 980        """
 981        if os.name == "nt":  # Windows
 982            # Handle special quoting and escaping for Windows
 983            quoted_args = []
 984            for arg in command_array:
 985                if "\n" in arg:
 986                    # Replace newlines with literal \n
 987                    arg = arg.replace("\n", "\\n")
 988                # Escape double quotes and wrap in double quotes if necessary
 989                if " " in arg or any(ch in arg for ch in ("&", "^", "%", "$", "#", '"')):
 990                    arg = '"{}"'.format(arg.replace('"', '\\"'))
 991                quoted_args.append(arg)
 992            command_string = "{}".format(" ".join(quoted_args))
 993        else:  # POSIX (Linux/macOS)
 994            # Use shlex.quote for safe quoting
 995            # quoted_args = [shlex.quote(arg) for arg in arguments]
 996            command_string = shlex.join(command_array)
 997
 998        with open(filename, "w", encoding="utf-8") as f:
 999            f.write(command_string + "\n")
1000
1001        return filename
1002
1003    def convert_bytes(self, bytes) -> str:
1004        # List of units in order
1005        units = ["B", "KB", "MB", "GB", "TB", "PB", "EB"]
1006
1007        # Start with bytes and convert to larger units
1008        unit_index = 0
1009        while bytes >= 1024 and unit_index < len(units) - 1:
1010            bytes /= 1024
1011            unit_index += 1
1012
1013        # Format and return the result
1014        return f"{bytes:.2f} {units[unit_index]}"
1015
1016    def print_stats(self, options) -> None:
1017        if options.get("stats_as_json", False):
1018            # \033[K clears the line after printing the JSON
1019            return
1020
1021        if self.logger.getEffectiveLevel() > logging.INFO:
1022            return
1023
1024        if not (self.stats.get("video") or self.stats.get("audio")):
1025            return
1026
1027        # Build the output parts in a list
1028        parts = [f"{self.stats.get('id')}:"]
1029
1030        if self.stats.get("video"):
1031            v = self.stats.get("video", {})
1032            v_str = f"Video: {v.get('downloaded_segments', 0)}/{v.get('latest_sequence', 0)}"
1033            if v.get("status"):
1034                v_str += f" ({v.get('status').capitalize()})"
1035            parts.append(v_str)
1036
1037        if self.stats.get("audio"):
1038            a = self.stats.get("audio", {})
1039            a_str = f"Audio: {a.get('downloaded_segments', 0)}/{a.get('latest_sequence', 0)}"
1040            # Note: Fixed a likely typo in your original code where you checked
1041            # video status while printing audio stats
1042            if a.get("status"):
1043                a_str += f" ({a.get('status').capitalize()})"
1044            parts.append(a_str)
1045
1046        if self.stats.get("video", {}).get("current_filesize") or self.stats.get("audio", {}).get("current_filesize"):
1047            size = self.stats.get("video", {}).get("current_filesize", 0) + self.stats.get("audio", {}).get(
1048                "current_filesize", 0
1049            )
1050            parts.append(f"~{self.convert_bytes(size)} downloaded")
1051
1052        # Join everything with commas or spaces
1053        " ".join(parts)
1054
1055        if options.get("new_line", False):
1056            pass
1057        else:
1058            # \r moves to start, \033[K clears anything left over from the previous longer line
1059            pass
1060
1061    def add_url_param(self, url: str, key, value) -> str:
1062        parsed = urlparse(url)
1063        query = parse_qs(parsed.query)
1064        query[key] = [value]  # add or replace parameter
1065
1066        new_query = urlencode(query, doseq=True)
1067        new_url = parsed._replace(query=new_query)
1068        return str(urlunparse(new_url))
1069
1070    def trim_info_json(self, info_dict: dict, keys_to_keep: set):
1071        return {k: info_dict[k] for k in info_dict.keys() & keys_to_keep}
1072
1073    def remove_format_segment_playlist_from_info_dict(self, info_dict: dict) -> None:
1074        for stream_format in info_dict.get("formats", []):
1075            with contextlib.suppress(KeyError, AttributeError, TypeError):
1076                stream_format.pop("fragments", None)
1077
1078    def refresh_info_json(
1079        self,
1080        update_threshold: float,
1081        video_id,
1082        cookies=None,
1083        additional_options=None,
1084        include_dash=False,
1085        include_m3u8=False,
1086    ):
1087        # Check if time difference is greater than the threshold. If doesn't exist, subtraction of zero will always be true
1088        if time.time() - self.refresh_json.get("refresh_time", 0.0) > update_threshold:
1089            self.refresh_json, self.live_status = get_video_info(
1090                video_id=video_id,
1091                wait=False,
1092                cookies=cookies,
1093                additional_options=additional_options,
1094                include_dash=include_dash,
1095                include_m3u8=include_m3u8,
1096                clean_info_dict=True,
1097            )
1098
1099            # Remove unnecessary items for info.json used purely for url refresh
1100            self.remove_format_segment_playlist_from_info_dict(self.refresh_json)
1101
1102            self.refresh_json.pop("thumbnails", None)
1103            self.refresh_json.pop("tags", None)
1104            self.refresh_json.pop("description", None)
1105
1106            # Add refresh time for reference
1107            self.refresh_json["refresh_time"] = time.time()
1108
1109            return self.refresh_json, self.live_status
1110        return self.refresh_json, self.live_status
LiveStreamDownloader( kill_all: threading.Event = <threading.Event at 0x7ff6cb58a210: unset>, cleanup: threading.Event = <threading.Event at 0x7ff6cb58a990: unset>, logger: logging.Logger | logging.LoggerAdapter | None = None, kill_this: threading.Event | None = None)
40    def __init__(
41        self,
42        kill_all: threading.Event = threading.Event(),
43        cleanup: threading.Event = threading.Event(),
44        logger: logging.Logger | logging.LoggerAdapter | None = None,
45        kill_this: threading.Event | None = None,
46    ) -> None:
47        if logger:
48            self.logger = logger
49        else:
50            # 1. Create a Named Logger instance and assign it to self.logger
51            self.logger = logging.getLogger(self.__class__.__name__)
52            # Ensure it processes messages (important if you don't call setup_logging immediately)
53        if isinstance(self.logger, logging.Logger):
54            self.logger.propagate = False
55        self.kill_this = kill_this or threading.Event()
56
57        # Global state converted to instance attributes
58        self.kill_all = kill_all
59        self.cleanup = cleanup
60        self.live_chat_result = None
61        self.chat_timeout = None
62        # File name dictionary
63        self.file_names = {"databases": [], "streams": {}}
64        self.stats = {}
65
66        self.refresh_json = {}
67        self.live_status = ""
68        self.lock: threading.Lock = threading.Lock()
kill_this
kill_all
cleanup
live_chat_result
chat_timeout
file_names
stats
refresh_json
live_status
lock: _thread.lock
def download_stream( self, info_dict, resolution, batch_size=5, max_workers=1, folder=None, file_name=None, keep_database=False, cookies=None, retries=5, yt_dlp_options=None, proxies=None, yt_dlp_sort=None, include_dash=False, include_m3u8=False, force_m3u8=False, manifest=0, **kwargs):
 71    def download_stream(
 72        self,
 73        info_dict,
 74        resolution,
 75        batch_size=5,
 76        max_workers=1,
 77        folder=None,
 78        file_name=None,
 79        keep_database=False,
 80        cookies=None,
 81        retries=5,
 82        yt_dlp_options=None,
 83        proxies=None,
 84        yt_dlp_sort=None,
 85        include_dash=False,
 86        include_m3u8=False,
 87        force_m3u8=False,
 88        manifest=0,
 89        **kwargs,
 90    ):
 91        try:
 92            download_params = locals().copy()
 93            download_params.update({"download_function": self.download_stream})
 94            file = None
 95            filetype = None
 96
 97            with DownloadStream(
 98                info_dict,
 99                resolution=resolution,
100                batch_size=batch_size,
101                max_workers=max_workers,
102                folder=folder,
103                file_name=file_name,
104                cookies=cookies,
105                fragment_retries=retries,
106                yt_dlp_options=yt_dlp_options,
107                proxies=proxies,
108                yt_dlp_sort=yt_dlp_sort,
109                include_dash=include_dash,
110                include_m3u8=include_m3u8,
111                force_m3u8=force_m3u8,
112                download_params=download_params,
113                livestream_coordinator=self,
114                **kwargs,
115            ) as downloader:
116                self.stats["status"] = "Recording"
117                downloader.live_dl()
118                file_name = downloader.combine_segments_to_file(downloader.merged_file_name)
119                if not keep_database:
120                    self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}")
121                    downloader.delete_temp_database()
122                elif downloader.temp_db_file != ":memory:":
123                    database_file = FileInfo(downloader.temp_db_file, file_type="database", format=downloader.format)
124                    self.file_names["databases"].append(database_file)
125
126                file = FileInfo(file_name, file_type=downloader.type, format=downloader.format)
127                filetype = downloader.type
128
129            self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file})
130
131            return file, filetype
132        except Exception:
133            self.logger.exception("Unexpected error occurred while downloading stream")
134            raise
def download_stream_direct( self, info_dict, resolution, batch_size, max_workers, folder=None, file_name=None, keep_state=False, cookies=None, retries=5, yt_dlp_options=None, proxies=None, yt_dlp_sort=None, include_dash=False, include_m3u8=False, force_m3u8=False, manifest=0, **kwargs):
137    def download_stream_direct(
138        self,
139        info_dict,
140        resolution,
141        batch_size,
142        max_workers,
143        folder=None,
144        file_name=None,
145        keep_state=False,
146        cookies=None,
147        retries=5,
148        yt_dlp_options=None,
149        proxies=None,
150        yt_dlp_sort=None,
151        include_dash=False,
152        include_m3u8=False,
153        force_m3u8=False,
154        manifest=0,
155        **kwargs,
156    ):
157        try:
158            download_params = locals().copy()
159            download_params.update({"download_function": self.download_stream_direct})
160            file = None
161            filetype = None
162
163            with DownloadStreamDirect(
164                info_dict,
165                resolution=resolution,
166                max_workers=max_workers,
167                folder=folder,
168                file_name=file_name,
169                cookies=cookies,
170                fragment_retries=retries,
171                yt_dlp_options=yt_dlp_options,
172                proxies=proxies,
173                yt_dlp_sort=yt_dlp_sort,
174                include_dash=include_dash,
175                include_m3u8=include_m3u8,
176                force_m3u8=force_m3u8,
177                download_params=download_params,
178                livestream_coordinator=self,
179                **kwargs,
180            ) as downloader:
181                self.stats["status"] = "Recording"
182                file_name = downloader.live_dl()
183                file = FileInfo(file_name, file_type=downloader.type, format=downloader.format)
184                filetype = downloader.type
185                downloader.delete_state_file()
186
187            self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file})
188            return file, filetype
189        except Exception:
190            self.logger.exception("Unexpected error occurred while downloading stream")
191            raise
def recover_stream( self, info_dict, resolution, batch_size=5, max_workers=5, folder=None, file_name=None, keep_database=False, cookies=None, retries=5, yt_dlp_options=None, proxies=None, yt_dlp_sort=None, force_merge=False, recovery_failure_tolerance=0, manifest=0, stream_urls: list | None = None, no_merge=False, **kwargs):
193    def recover_stream(
194        self,
195        info_dict,
196        resolution,
197        batch_size=5,
198        max_workers=5,
199        folder=None,
200        file_name=None,
201        keep_database=False,
202        cookies=None,
203        retries=5,
204        yt_dlp_options=None,
205        proxies=None,
206        yt_dlp_sort=None,
207        force_merge=False,
208        recovery_failure_tolerance=0,
209        manifest=0,
210        stream_urls: list | None = None,
211        no_merge=False,
212        **kwargs,
213    ):
214        if stream_urls is None:
215            stream_urls = []
216        try:
217            file = None
218            filetype = None
219
220            with StreamRecovery(
221                info_dict,
222                resolution=resolution,
223                batch_size=batch_size,
224                max_workers=max_workers,
225                folder=folder,
226                file_name=file_name,
227                cookies=cookies,
228                fragment_retries=retries,
229                proxies=proxies,
230                yt_dlp_sort=yt_dlp_sort,
231                livestream_coordinator=self,
232                stream_urls=stream_urls,
233                **kwargs,
234            ) as downloader:
235                self.stats["status"] = "Recording"
236                result = downloader.live_dl()
237                if (force_merge or result <= 0 or result <= recovery_failure_tolerance) and (not no_merge):
238                    if result > 0:
239                        self.logger.warning(
240                            f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting"
241                        )
242                    file_name = downloader.combine_segments_to_file(downloader.merged_file_name)
243                    if not keep_database:
244                        self.logger.info(f"Merging to ts complete, removing {downloader.temp_db_file}")
245                        downloader.delete_temp_database()
246                    elif downloader.temp_db_file != ":memory:":
247                        database_file = FileInfo(
248                            downloader.temp_db_file, file_type="database", format=downloader.format
249                        )
250                        self.file_names["databases"].append(database_file)
251                else:
252                    msg = f"({downloader.id}) Stream recovery of format {downloader.format} has {result} outstanding segments which were not able to complete. Exitting"
253                    raise VideoDownloadError(msg)
254                file = FileInfo(file_name, file_type=downloader.type, format=downloader.format)
255                filetype = downloader.type
256
257            self.file_names.setdefault("streams", {}).setdefault(manifest, {}).update({str(filetype).lower(): file})
258        except Exception:
259            self.logger.exception("Unexpected error occurred while downloading stream")
260            raise
261        return file, filetype
def submit_download( self, executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False):
263    def submit_download(
264        self, executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False
265    ):
266        extra_kwargs = {}
267
268        # Options only not used by recovery
269        if not options.get("recovery", False):
270            extra_kwargs.update(
271                {
272                    "include_dash": options.get("dash", False),
273                    "include_m3u8": options.get("m3u8", False),
274                    "force_m3u8": options.get("force_m3u8", False),
275                }
276            )
277
278        # Select the function
279        if options.get("recovery", False):
280            func = self.recover_stream
281            extra_kwargs.update(
282                {
283                    "force_merge": options.get("force_recovery_merge", False),
284                    "recovery_failure_tolerance": options.get("recovery_failure_tolerance", 0),
285                }
286            )
287        elif options.get("direct_to_ts", False):
288            func = self.download_stream_direct
289            extra_kwargs.update(
290                {
291                    "keep_state": options.get("keep_temp_files", False) or options.get("keep_database_file", False),
292                }
293            )
294        else:
295            func = self.download_stream
296            extra_kwargs.update(
297                {
298                    "keep_database": options.get("keep_temp_files", False) or options.get("keep_database_file", False),
299                }
300            )
301
302        kwargs = {
303            "info_dict": info_dict,
304            "resolution": "audio_only" if is_audio else resolution,
305            "batch_size": options.get("batch_size", 1),
306            "max_workers": options.get("threads", 1),
307            "folder": download_folder,
308            "file_name": file_name,
309            "retries": options.get("segment_retries"),
310            "cookies": options.get("cookies"),
311            "yt_dlp_options": options.get("ytdlp_options", None),
312            "proxies": options.get("proxy", None),
313            "yt_dlp_sort": options.get("custom_sort", None),
314        }
315
316        if extra_kwargs:
317            kwargs.update(extra_kwargs)
318
319        # Add any extra values not yet existing
320        for key, value in options.items():
321            kwargs.setdefault(key, value)
322
323        self.logger.debug(f"Starting executor with: {json.dumps(kwargs)}")
324
325        # Submit to executor
326        future = executor.submit(func, **kwargs)
327        futures.add(future)
328        return future
def download_segments( self, info_dict, resolution='best', options=None, thread_event: threading.Event | None = None) -> None:
331    def download_segments(
332        self, info_dict, resolution="best", options=None, thread_event: threading.Event | None = None
333    ) -> None:
334        if options is None:
335            options = {}
336        futures = set()
337
338        if thread_event is not None:
339            self.kill_all = thread_event
340
341        self.stats["id"] = info_dict.get("id", None)
342        self.stats["status"] = "Starting"
343        self.logger.debug(json.dumps(options, indent=4))
344        outputFile = self.output_filename(info_dict=info_dict, outtmpl=options.get("output"))
345        file_name = None
346        # Requires testing
347        if options.get("temp_folder") is not None and options.get("temp_folder") != os.path.dirname(outputFile):
348            _output_folder, file_name = os.path.split(outputFile)
349            download_folder = self.output_filename(info_dict=info_dict, outtmpl=options.get("temp_folder"))
350            options["temp_folder"] = download_folder
351        else:
352            download_folder, file_name = os.path.split(outputFile)
353        options["filename"] = file_name
354        if download_folder:
355            os.makedirs(download_folder, exist_ok=True)
356
357        with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
358            done = not_done = set()
359            try:
360                # Download auxiliary files (thumbnail, info,json etc)
361                auxiliary_thread = executor.submit(self.download_auxiliary_files, info_dict=info_dict, options=options)
362                futures.add(auxiliary_thread)
363
364                live_chat_thread = None
365                if options.get("live_chat", False) is True:
366                    live_chat_thread = threading.Thread(
367                        target=self.download_live_chat, args=(info_dict, options), daemon=True
368                    )
369                    live_chat_thread.start()
370
371                format_check = Formats().getFormatURL(
372                    info_json=info_dict,
373                    resolution=resolution,
374                    sort=options.get("custom_sort"),
375                    include_dash=(options.get("dash", False) and not options.get("recovery", False)),
376                    include_m3u8=options.get("m3u8", False),
377                    force_m3u8=options.get("force_m3u8", False),
378                )
379                if not format_check:
380                    msg = "Resolution is not valid or does not exist in stream"
381                    raise ValueError(msg)
382                # For use of specificed format. Expects two values, but can work with more
383                # Video + Audio
384
385                if resolution.lower() not in ("audio_only", "ba"):
386                    # Video
387                    self.submit_download(
388                        executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=False
389                    )
390                    if format_check.protocol != "m3u8_native":
391                        # Audio
392                        self.submit_download(
393                            executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True
394                        )
395                # Audio-only
396                else:
397                    self.submit_download(
398                        executor, info_dict, resolution, options, download_folder, file_name, futures, is_audio=True
399                    )
400
401                while True:
402                    if self.kill_all.is_set() or self.kill_this.is_set():
403                        msg = "Thread kill event is set, ending..."
404                        raise KeyboardInterrupt(msg)
405                    done, not_done = concurrent.futures.wait(
406                        futures, timeout=1, return_when=concurrent.futures.FIRST_COMPLETED
407                    )
408                    # Continuously check for completion or interruption
409                    for future in done:
410                        result, type = future.result()
411                        self.logger.info(f"\033[31m{result}\033[0m")
412
413                        if type == "auxiliary":
414                            self.file_names.update(result)
415
416                            # Remove parts that would not be needed by stream downloaders
417                            self.remove_format_segment_playlist_from_info_dict(info_dict=info_dict)
418                            info_dict.pop("thumbnails", None)
419                            # Remove no longer required parts for the rest of the downloader
420                            # info_dict = self.trim_info_json(info_dict=info_dict, keys_to_keep={"id", "ext", "upload_date", "original_url", "description", "fulltitle", "channel"})
421
422                        elif str(type).lower() == "video":
423                            # self.file_names['video'] = result
424                            pass
425                        elif str(type).lower() == "audio":
426                            # self.file_names['audio'] = result
427                            pass
428                        elif str(type).lower() == "live_chat":
429                            pass
430                        else:
431                            self.file_names[str(type)] = result
432
433                        futures.discard(future)
434
435                    if len(not_done) <= 0:
436                        break
437                    # else:
438                    #    time.sleep(0.9)
439                    self.print_stats(options=options)
440
441                if live_chat_thread and live_chat_thread.is_alive():
442                    self.logger.info("Media downloads finished, waiting for live chat to end...")
443                    if options.get("stop_chat_when_done"):
444                        self.chat_timeout = time.time()
445                    while live_chat_thread.is_alive() and not (self.kill_all.is_set() or self.kill_this.is_set()):
446                        time.sleep(0.1)
447
448            except KeyboardInterrupt:
449                self.kill_this.set()
450                self.logger.debug("Keyboard interrupt detected")
451                if len(not_done) > 0:
452                    self.logger.debug("Cancelling remaining threads")
453                for future in not_done:
454                    _ = future.cancel()
455                done, not_done = concurrent.futures.wait(
456                    futures, timeout=5, return_when=concurrent.futures.ALL_COMPLETED
457                )
458                executor.shutdown(wait=False, cancel_futures=True)
459                self.logger.debug("Shutdown all threads")
460                self.stats["status"] = "Cancelled"
461                raise
462
463        self.stats["status"] = "Muxing"
464        self.create_mp4(file_names=self.file_names, info_dict=info_dict, options=options)
465
466        self.stats["status"] = "Moving"
467        self.move_to_final(options=options, output_file=outputFile, file_names=self.file_names)
468        self.stats["status"] = "Finished"
469        # move_to_final(info_dict, options, file_names)
def output_filename(self, info_dict, outtmpl):
471    def output_filename(self, info_dict, outtmpl):
472        return str(YoutubeDL().prepare_filename(info_dict, outtmpl=outtmpl)).replace(
473            "%", "%"
474        )  # Replace normal percent sign with unicode version to prevent ffmpeg errors
def move_to_final(self, options, output_file, file_names) -> None:
476    def move_to_final(self, options, output_file, file_names) -> None:
477        def maybe_move(key, dest_func, file_names_dict=file_names, option_flag=None) -> None:
478            """key: key in file_names
479            dest_func: func -> pathlib -> string dest path
480            delete_if_false: if True, delete when option_flag not set
481            option_flag: name of boolean option (write_thumbnail etc)
482            """
483            f = file_names_dict.get(key)
484            if not f:
485                return
486            try:
487                # deletion case (thumbnail / ffmpeg)
488                if option_flag is not None and not options.get(option_flag, False):
489                    self.logger.info(f"Removing {f.absolute()}")
490                    f.unlink(missing_ok=True)
491                    file_names_dict.pop(key, None)
492                    return
493
494                dest = dest_func(f)
495                if str(f).strip() != str(dest).strip():
496                    self.logger.info(f"Moving {f.absolute()} → {dest}")
497                    shutil.move(f.absolute(), dest)
498                else:
499                    self.logger.debug(f"{f.absolute()} is already in final destination")
500            except Exception as e:
501                self.logger.exception(f"unable to move {key}: {e}")
502
503        # ensure output dir exists
504        out_dir = os.path.dirname(output_file)
505        if out_dir:
506            os.makedirs(out_dir, exist_ok=True)
507
508        # === individual file handlers ===
509
510        maybe_move("thumbnail", lambda f: f"{output_file}{f.suffix}", option_flag="write_thumbnail")
511
512        maybe_move(
513            "info_json",
514            lambda f: f"{output_file}.info.json",
515        )
516
517        maybe_move(
518            "description",
519            lambda f: f"{output_file}{f.suffix}",
520        )
521
522        stream_manifests = list(self.file_names["streams"].items())
523        for manifest, stream in stream_manifests:
524            stream_output_file = output_file
525            if len(stream_manifests) > 1:
526                stream_output_file = f"{output_file}.{manifest}"
527            maybe_move("video", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream)
528
529            maybe_move("audio", lambda f: f"{stream_output_file}.{f._format}{f.suffix}", file_names_dict=stream)
530
531            maybe_move("merged", lambda f: f"{stream_output_file}{f.suffix}", file_names_dict=stream)
532
533        maybe_move(
534            "live_chat",
535            lambda f: f"{output_file}.live_chat.zip",
536        )
537
538        # special: databases = list
539        try:
540            for f in file_names.get("databases", []):
541                dest = f"{output_file}.{f._format}{f.suffix}"
542                if str(f).strip() != str(dest).strip():
543                    self.logger.info(f"Moving {f.absolute()} → {dest}")
544                    shutil.move(f.absolute(), dest)
545        except Exception as e:
546            self.logger.exception(f"unable to move database files: {e}")
547
548        # special: ffmpeg_cmd
549        maybe_move("ffmpeg_cmd", lambda f: f"{output_file}.ffmpeg.txt", option_flag="write_ffmpeg_command")
550
551        # remove temp folder
552        """
553        if options.get('temp_folder', None) is not None:
554            try:
555                os.rmdir(options.get('temp_folder'))
556            except OSError as e:
557                if e.errno == errno.ENOTEMPTY:
558                    self.logger.warning(f"Error: Directory not empty: {e.filename}")
559                else:
560                    self.logger.exception(f"Error removing temp folder: {e}")
561            except Exception as e:
562                self.logger.exception(f"Error removing temp folder: {e}")
563        """
564        self.logger.info("Finished moving files from temporary directory to output destination")
def download_live_chat(self, info_dict, options):
566    def download_live_chat(self, info_dict, options):
567        filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id")
568
569        base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename
570
571        class YTDLP_Chat_logger(YTDLPLogger):
572            def __init__(self, logger: logging.Logger | logging.LoggerAdapter | None = None) -> None:
573                if logger is None:
574                    logger = logging.getLogger()
575                super().__init__(logger=logger)
576
577            def prefix(self, msg: str):
578                if msg.startswith("[live-chat] "):
579                    return msg
580                return f"[live-chat] {msg}"
581
582            def debug(self, msg) -> None:
583                super().debug(self.prefix(msg))
584
585            def info(self, msg) -> None:
586                super().info(self.prefix(msg))
587
588            def warning(self, msg) -> None:
589                super().warning(self.prefix(msg))
590
591            def error(self, msg) -> None:
592                super().error(self.prefix(msg))
593
594        logger = self.logger
595
596        ydl_opts = {
597            "skip_download": True,  # Skip downloading video/audio
598            "logger": YTDLP_Chat_logger(logger=logger),
599            #'quiet': True,
600            "cookiefile": options.get("cookies", None),
601            "retries": 25,
602            "concurrent_fragment_downloads": 3,
603            #'live_from_start': True,
604            "writesubtitles": True,  # Extract subtitles (live chat)
605            "subtitlesformat": "json",  # Set format to JSON
606            "subtitleslangs": ["live_chat"],  # Only extract live chat subtitles
607            "outtmpl": base_output,  # Save to a JSON file
608        }
609        self.logger.debug(options.get("ytdlp_options", {}))
610        ydl_opts.update(options.get("ytdlp_options", {}))
611
612        livechat_filename = base_output + ".live_chat.json"
613
614        self.logger.info(f"Downloading live chat to: {livechat_filename}")
615
616        # Download live chat using yt-dlp
617        with YoutubeDL(ydl_opts) as ydl:
618            try:
619                ydl.download([info_dict.get("original_url", info_dict.get("webpage_url", info_dict.get("url")))])
620            except Exception as e:
621                self.logger.exception(f"Failed to download live chat: {e}")
622
623        time.sleep(1)
624        part_file = f"{livechat_filename}.part"
625        if os.path.exists(part_file):
626            # Append part to
627            chunk_size = 1024 * 1024 * 10  # number of characters per chunk, up to 10M characters
628            with open(part_file, encoding="utf-8") as fa, open(livechat_filename, "a", encoding="utf-8") as fb:
629                while chunk := fa.read(chunk_size):
630                    fb.write(chunk)
631            os.remove(f"{livechat_filename}.part")
632
633        try:
634            if not os.path.exists(livechat_filename):
635                self.logger.warning(f"Live chat file not found: {livechat_filename}")
636                return None, None
637
638            zip_filename = base_output + ".live_chat.zip"
639            with zipfile.ZipFile(
640                zip_filename, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=9, allowZip64=True
641            ) as zipf:
642                zipf.write(livechat_filename, arcname=os.path.basename(livechat_filename))
643            os.remove(livechat_filename)
644            live_chat = {"live_chat": FileInfo(zip_filename, file_type="live_chat")}
645            self.file_names.update(live_chat)
646            return live_chat, "live_chat"
647        except Exception as e:
648            self.logger.exception(f"\033[31m{e}\033[0m")
def replace_ip_in_json(self, file_name) -> None:
650    def replace_ip_in_json(self, file_name) -> None:
651        pattern = re.compile(r"((?:[0-9]{1,3}\.){3}[0-9]{1,3})|((?:[a-f0-9]{1,4}:){7}[a-f0-9]{1,4})")
652
653        with open(file_name, encoding="utf-8") as file:
654            content = file.read()
655
656        modified_content = re.sub(pattern, "0.0.0.0", content)
657
658        with open(file_name, "w", encoding="utf-8") as file:
659            file.write(modified_content)
def remove_urls_from_json(self, file_name) -> None:
661    def remove_urls_from_json(self, file_name) -> None:
662        with open(file_name, encoding="utf-8") as file:
663            data = json.load(file)
664
665        if data.get("formats", None) is not None:
666            for format in data["formats"]:
667                if format.get("url") is not None:
668                    format["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
669
670                if format.get("manifest_url") is not None:
671                    format["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
672
673                format.pop("fragment_base_url", None)
674                format.pop("fragments", None)
675
676        if data.get("thumbnails", None) is not None:
677            for thumbnail in data["thumbnails"]:
678                if thumbnail.get("url", None) is not None:
679                    parsed_url = urlparse(thumbnail.get("url", ""))
680                    thumbnail["url"] = f"{parsed_url.scheme}://{parsed_url.netloc}{parsed_url.path}"
681
682        if data.get("url", None) is not None:
683            data["url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
684
685        if data.get("manifest_url", None) is not None:
686            data["manifest_url"] = "https://www.youtube.com/watch?v={}".format(data.get("id", ""))
687
688        data["removed_urls"] = True
689
690        with open(file_name, "w", encoding="utf-8") as file:
691            json.dump(data, file)
def download_auxiliary_files(self, info_dict, options):
693    def download_auxiliary_files(self, info_dict, options):
694        filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id")
695
696        base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename
697
698        created_files = {}
699
700        logger = self.logger
701
702        class YTDLP_Auxiliary_logger(YTDLPLogger):
703            def __init__(self, logger: logging.Logger | None = None) -> None:
704                if logger is None:
705                    logger = logging.getLogger()
706                super().__init__(logger=logger)
707
708            def prefix(self, msg: str):
709                if msg.startswith("[live-chat] "):
710                    return msg
711                return f"[live-chat] {msg}"
712
713            def debug(self, msg) -> None:
714                super().debug(self.prefix(msg))
715
716            def info(self, msg) -> None:
717                super().info(self.prefix(msg))
718
719            def warning(self, msg) -> None:
720                super().warning(self.prefix(msg))
721
722            def error(self, msg) -> None:
723                super().error(self.prefix(msg))
724
725        ydl_opts = {
726            "skip_download": True,
727            "quiet": True,
728            #        'cookiefile': options.get('cookies', None),
729            "writeinfojson": options.get("write_info_json", False),
730            "writedescription": options.get("write_description", False),
731            "writethumbnail": (options.get("write_thumbnail", False) or options.get("embed_thumbnail", False)),
732            "outtmpl": base_output,
733            "retries": 10,
734            "logger": YTDLPLogger(logger=logger),
735        }
736        if options.get("proxy", None) is not None:
737            ydl_opts["proxy"] = next(iter((options.get("proxy", None) or {}).values()), None)
738
739        with YoutubeDL(ydl_opts) as ydl:
740            # base_name = ydl.prepare_filename(info_dict)
741            # result = ydl.download_with_info_file(info_dict)
742
743            if ydl._write_info_json("video", info_dict, ydl.prepare_filename(info_dict, "infojson")) or os.path.exists(
744                ydl.prepare_filename(info_dict, "infojson")
745            ):
746                created_files["info_json"] = FileInfo(
747                    ydl.prepare_filename(info_dict, "infojson"), file_type="info_json"
748                )
749                try:
750                    if options.get("remove_ip_from_json"):
751                        self.replace_ip_in_json(created_files["info_json"].absolute())
752                    if options.get("clean_urls"):
753                        self.remove_urls_from_json(created_files["info_json"].absolute())
754                except Exception as e:
755                    self.logger.exception(str(e))
756
757            if ydl._write_description(
758                "video", info_dict, ydl.prepare_filename(info_dict, "description")
759            ) or os.path.exists(ydl.prepare_filename(info_dict, "description")):
760                created_files["description"] = FileInfo(
761                    ydl.prepare_filename(info_dict, "description"), file_type="description"
762                )
763
764            thumbnails = ydl._write_thumbnails("video", info_dict, ydl.prepare_filename(info_dict, "thumbnail"))
765
766            if thumbnails:
767                created_files["thumbnail"] = FileInfo(thumbnails[0][0], file_type="thumbnail")
768
769        return created_files, "auxiliary"
def create_mp4(self, file_names, info_dict, options):
771    def create_mp4(self, file_names, info_dict, options):
772        # self.logger.debug("Files: {0}".format(json.dumps(file_names)))
773        stream_manifests = list(self.file_names["streams"].items())
774        for manifest, stream in stream_manifests:
775            index = 0
776            thumbnail = None
777            video = None
778            ext = options.get("ext", None)
779
780            ffmpeg_builder = ["ffmpeg", "-y", "-hide_banner", "-nostdin", "-loglevel", "error", "-stats"]
781
782            # Add input files
783            if stream.get("video", None):
784                input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("video").absolute())]
785                ffmpeg_builder.extend(input)
786                video = index
787                index += 1
788
789            if stream.get("audio", None):
790                input = ["-thread_queue_size", "1024", "-seekable", "0", "-i", str(stream.get("audio").absolute())]
791                ffmpeg_builder.extend(input)
792                index += 1
793                if video is None and ext is None:
794                    ext = ".mka"
795
796            # Determine output path
797            filename = options.get("filename") if options.get("filename") is not None else info_dict.get("id")
798
799            base_output = os.path.join(options.get("temp_folder"), filename) if options.get("temp_folder") else filename
800
801            if len(stream_manifests) > 1:
802                base_output = f"{base_output}.{manifest}"
803
804            if ext is None:
805                self.logger.debug("No extension detected, switching to yt-dlp decided video (defaulting to MP4)")
806                ext = info_dict.get("ext", ".mp4")
807            if ext is not None and not str(ext).startswith("."):
808                ext = "." + str(ext)
809            if not base_output.endswith(ext):
810                base_output = base_output + ext
811
812            if file_names.get("thumbnail", None) and options.get("embed_thumbnail", True):
813                if file_names.get("thumbnail").exists():
814                    # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version
815                    guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type)
816
817                    mime_type, _ = guess(file_names.get("thumbnail"))
818
819                    # If not jpeg or png, convert to png
820                    if str(mime_type) not in ("image/jpeg", "image/png"):
821                        self.logger.info(
822                            "{} is not a JPG or PNG file, converting to png".format(file_names.get("thumbnail").name)
823                        )
824                        png_thumbnail = file_names.get("thumbnail").with_suffix(".png")
825
826                        thumbnail_conversion = [
827                            "ffmpeg",
828                            "-y",
829                            "-hide_banner",
830                            "-nostdin",
831                            "-loglevel",
832                            "error",
833                            "-stats",
834                            "-i",
835                            str(file_names.get("thumbnail").absolute()),
836                            "-c",
837                            "png",
838                            "-compression_level",
839                            "9",
840                            "-pred",
841                            "mixed",
842                            str(png_thumbnail.absolute()),
843                        ]
844                        try:
845                            result = subprocess.run(
846                                thumbnail_conversion,
847                                capture_output=True,
848                                text=True,
849                                encoding="utf-8",
850                                check=True,
851                            )
852                            self.logger.debug("Replacing thumbnail with .png version")
853                            file_names.pop("thumbnail").unlink(missing_ok=True)
854                            file_names["thumbnail"] = FileInfo(png_thumbnail, file_type="thumbnail")
855
856                        except subprocess.CalledProcessError as e:
857                            self.logger.exception(e.stderr)
858                            self.logger.critical(e)
859
860                    thumbnail = index
861                    if ext.lower() != ".mkv":  # Don't add input file for mkv, use attach later
862                        input = [
863                            "-thread_queue_size",
864                            "1024",
865                            "-seekable",
866                            "0",
867                            "-i",
868                            str(file_names.get("thumbnail").absolute()),
869                        ]
870                        ffmpeg_builder.extend(input)
871                        index += 1
872                else:
873                    self.logger.error(
874                        "Thumbnail file: {} is missing, continuing without embedding".format(
875                            file_names.get("thumbnail").absolute()
876                        )
877                    )
878
879            # Add faststart
880            ffmpeg_builder.extend(["-movflags", "faststart"])
881
882            # Add mappings
883            for i in range(index):
884                input = ["-map", str(i)]
885                ffmpeg_builder.extend(input)
886
887            if thumbnail is not None:
888                if ext.lower() == ".mkv":  # If file will be mkv, attach file instead
889                    # Use "guess_file_type" if function exists (added in 3.13), otherwise fall back to depreciated version
890                    guess = getattr(mimetypes, "guess_file_type", mimetypes.guess_type)
891                    mime_type, _ = guess(file_names.get("thumbnail"))
892                    ffmpeg_builder.extend(
893                        [
894                            "-attach",
895                            str(file_names.get("thumbnail").absolute()),
896                            "-metadata:s:t:0",
897                            "filename=cover{}".format(file_names.get("thumbnail").suffix),
898                            "-metadata:s:t:0",
899                            "mimetype={}".format(mime_type or "application/octet-stream"),
900                        ]
901                    )
902
903                else:  # For other formats, attach using disposition instead
904                    ffmpeg_builder.extend([f"-disposition:{thumbnail}", "attached_pic"])
905
906            # Add Copy codec
907            ffmpeg_builder.extend(["-c", "copy"])
908
909            # Add metadata
910            ffmpeg_builder.extend(["-metadata", "DATE={}".format(info_dict.get("upload_date"))])
911            ffmpeg_builder.extend(
912                [
913                    "-metadata",
914                    "COMMENT={}\n{}".format(info_dict.get("original_url"), info_dict.get("description", "")),
915                ]
916            )
917            ffmpeg_builder.extend(["-metadata", "TITLE={}".format(info_dict.get("fulltitle"))])
918            ffmpeg_builder.extend(["-metadata", "ARTIST={}".format(info_dict.get("channel"))])
919
920            # Add output file to ffmpeg command
921            ffmpeg_builder.append(str(Path(base_output).absolute()))
922
923            if options.get("write_ffmpeg_command", True):
924                ffmpeg_command_file = f"{filename}.ffmpeg.txt"
925                file_names["streams"][manifest]["ffmpeg_cmd"] = FileInfo(
926                    self.write_ffmpeg_command(ffmpeg_builder, ffmpeg_command_file), file_type="ffmpeg_command"
927                )
928
929            if not (options.get("merge", True)):
930                return file_names
931
932            self.logger.debug("FFmpeg command: {}".format(" ".join(ffmpeg_builder)))
933            self.logger.info(f"Executing ffmpeg. Outputting to {ffmpeg_builder[-1]}")
934            try:
935                result = subprocess.run(
936                    ffmpeg_builder,
937                    capture_output=True,
938                    text=True,
939                    encoding="utf-8",
940                    check=True,
941                )
942                self.logger.debug(f"FFmpeg STDOUT: {result.stdout}")
943                self.logger.debug(f"FFmpeg STDERR: {result.stderr}")
944            except subprocess.CalledProcessError as e:
945                self.logger.exception(e.stderr)
946                self.logger.critical(e)
947                raise
948
949            file_names["streams"][manifest]["merged"] = FileInfo(base_output, file_type="merged")
950            self.logger.info(
951                "Successfully merged files into: {}".format(file_names["streams"][manifest].get("merged").absolute())
952            )
953
954            # Remove temp video and audio files
955            if not (options.get("keep_ts_files") or options.get("keep_temp_files")):
956                if file_names["streams"][manifest].get("video"):
957                    self.logger.info("Removing {}".format(file_names["streams"][manifest].get("video").absolute()))
958                    file_names["streams"][manifest].get("video").unlink(missing_ok=True)
959                    file_names["streams"][manifest].pop("video", None)
960                if file_names["streams"][manifest].get("audio"):
961                    self.logger.info("Removing {}".format(file_names["streams"][manifest].get("audio").absolute()))
962                    file_names["streams"][manifest].get("audio").unlink(missing_ok=True)
963                    file_names["streams"][manifest].pop("audio", None)
964
965        return file_names
966        # for file in file_names:
967        #    os.remove(file)
def write_ffmpeg_command(self, command_array, filename):
 969    def write_ffmpeg_command(self, command_array, filename):
 970        # Determine the platform
 971        """
 972        Builds a platform-compatible FFmpeg command with proper quoting.
 973
 974        Args:
 975            command_array (list): List of arguments to append to the command.
 976            filename: Filename to write command to
 977
 978        Returns:
 979            str: A properly quoted FFmpeg command.
 980        """
 981        if os.name == "nt":  # Windows
 982            # Handle special quoting and escaping for Windows
 983            quoted_args = []
 984            for arg in command_array:
 985                if "\n" in arg:
 986                    # Replace newlines with literal \n
 987                    arg = arg.replace("\n", "\\n")
 988                # Escape double quotes and wrap in double quotes if necessary
 989                if " " in arg or any(ch in arg for ch in ("&", "^", "%", "$", "#", '"')):
 990                    arg = '"{}"'.format(arg.replace('"', '\\"'))
 991                quoted_args.append(arg)
 992            command_string = "{}".format(" ".join(quoted_args))
 993        else:  # POSIX (Linux/macOS)
 994            # Use shlex.quote for safe quoting
 995            # quoted_args = [shlex.quote(arg) for arg in arguments]
 996            command_string = shlex.join(command_array)
 997
 998        with open(filename, "w", encoding="utf-8") as f:
 999            f.write(command_string + "\n")
1000
1001        return filename

Builds a platform-compatible FFmpeg command with proper quoting.

Arguments:
  • command_array (list): List of arguments to append to the command.
  • filename: Filename to write command to
Returns:

str: A properly quoted FFmpeg command.

def convert_bytes(self, bytes) -> str:
1003    def convert_bytes(self, bytes) -> str:
1004        # List of units in order
1005        units = ["B", "KB", "MB", "GB", "TB", "PB", "EB"]
1006
1007        # Start with bytes and convert to larger units
1008        unit_index = 0
1009        while bytes >= 1024 and unit_index < len(units) - 1:
1010            bytes /= 1024
1011            unit_index += 1
1012
1013        # Format and return the result
1014        return f"{bytes:.2f} {units[unit_index]}"
def print_stats(self, options) -> None:
1016    def print_stats(self, options) -> None:
1017        if options.get("stats_as_json", False):
1018            # \033[K clears the line after printing the JSON
1019            return
1020
1021        if self.logger.getEffectiveLevel() > logging.INFO:
1022            return
1023
1024        if not (self.stats.get("video") or self.stats.get("audio")):
1025            return
1026
1027        # Build the output parts in a list
1028        parts = [f"{self.stats.get('id')}:"]
1029
1030        if self.stats.get("video"):
1031            v = self.stats.get("video", {})
1032            v_str = f"Video: {v.get('downloaded_segments', 0)}/{v.get('latest_sequence', 0)}"
1033            if v.get("status"):
1034                v_str += f" ({v.get('status').capitalize()})"
1035            parts.append(v_str)
1036
1037        if self.stats.get("audio"):
1038            a = self.stats.get("audio", {})
1039            a_str = f"Audio: {a.get('downloaded_segments', 0)}/{a.get('latest_sequence', 0)}"
1040            # Note: Fixed a likely typo in your original code where you checked
1041            # video status while printing audio stats
1042            if a.get("status"):
1043                a_str += f" ({a.get('status').capitalize()})"
1044            parts.append(a_str)
1045
1046        if self.stats.get("video", {}).get("current_filesize") or self.stats.get("audio", {}).get("current_filesize"):
1047            size = self.stats.get("video", {}).get("current_filesize", 0) + self.stats.get("audio", {}).get(
1048                "current_filesize", 0
1049            )
1050            parts.append(f"~{self.convert_bytes(size)} downloaded")
1051
1052        # Join everything with commas or spaces
1053        " ".join(parts)
1054
1055        if options.get("new_line", False):
1056            pass
1057        else:
1058            # \r moves to start, \033[K clears anything left over from the previous longer line
1059            pass
def add_url_param(self, url: str, key, value) -> str:
1061    def add_url_param(self, url: str, key, value) -> str:
1062        parsed = urlparse(url)
1063        query = parse_qs(parsed.query)
1064        query[key] = [value]  # add or replace parameter
1065
1066        new_query = urlencode(query, doseq=True)
1067        new_url = parsed._replace(query=new_query)
1068        return str(urlunparse(new_url))
def trim_info_json(self, info_dict: dict, keys_to_keep: set):
1070    def trim_info_json(self, info_dict: dict, keys_to_keep: set):
1071        return {k: info_dict[k] for k in info_dict.keys() & keys_to_keep}
def remove_format_segment_playlist_from_info_dict(self, info_dict: dict) -> None:
1073    def remove_format_segment_playlist_from_info_dict(self, info_dict: dict) -> None:
1074        for stream_format in info_dict.get("formats", []):
1075            with contextlib.suppress(KeyError, AttributeError, TypeError):
1076                stream_format.pop("fragments", None)
def refresh_info_json( self, update_threshold: float, video_id, cookies=None, additional_options=None, include_dash=False, include_m3u8=False):
1078    def refresh_info_json(
1079        self,
1080        update_threshold: float,
1081        video_id,
1082        cookies=None,
1083        additional_options=None,
1084        include_dash=False,
1085        include_m3u8=False,
1086    ):
1087        # Check if time difference is greater than the threshold. If doesn't exist, subtraction of zero will always be true
1088        if time.time() - self.refresh_json.get("refresh_time", 0.0) > update_threshold:
1089            self.refresh_json, self.live_status = get_video_info(
1090                video_id=video_id,
1091                wait=False,
1092                cookies=cookies,
1093                additional_options=additional_options,
1094                include_dash=include_dash,
1095                include_m3u8=include_m3u8,
1096                clean_info_dict=True,
1097            )
1098
1099            # Remove unnecessary items for info.json used purely for url refresh
1100            self.remove_format_segment_playlist_from_info_dict(self.refresh_json)
1101
1102            self.refresh_json.pop("thumbnails", None)
1103            self.refresh_json.pop("tags", None)
1104            self.refresh_json.pop("description", None)
1105
1106            # Add refresh time for reference
1107            self.refresh_json["refresh_time"] = time.time()
1108
1109            return self.refresh_json, self.live_status
1110        return self.refresh_json, self.live_status
__version__ = '0.0.0.post1+0bae4d5'