Merge branch 'master' of https://github.com/nstanger/process

nigel.stanger / process_podcast

Browse code Merge branch 'master' of https://github.com/nstanger/process_podcast master
commit 16761c7dc62b036decedd67c24529d198b9d2a85 2 parents 021b25e + 245da79 Nigel Stanger authored on 23 Aug 2021

Patch

Showing 3 changed files

Ignore Space Show notes View README.md
# process_podcast This script enables you to assemble an ingterated podcast of a lecture or class from several different inputs, including: * recorded audio * screen recordings * other video * JPEG images representing individual lecture slides * a PDF containing the lecture slides Audio and video may come from the same file, or from separate files. You also have the ability to insert small “joiner” sections of audio and video between major “segments” of the podcast. You can even configure the “joiner” so that it repeats the last frame of the previous segment for the desired duration. For help on command line options: `process_podcast.py --help`. For help on the podcast configuration file format, see the [configuration file documentation][]. ## Requirements * FFmpeg. * ImageMagick. * Python 3.5 or later with the following modules: * `pyparsing`; Linux: `pip install pyparsing`, `easy_install pyparsing`, or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-parsing`. Also see the [pyparsing documentation][]. * `pexpect`; Linux: `pip install pexpect`, `easy_install pexpect`, or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-pexpect`. Also see the [pexpect documentation][]. * There is a `requirements.txt` that you can use to quickly install all the required modules. [pyparsing documentation]: http://pyparsing.wikispaces.com/ "pyparsing documentation" [pexpect documentation]: https://pexpect.readthedocs.io/en/stable/ "pexpect documentation" [configuration file documentation]: config_help.md "configuration file documentation" ## Testing Run `python -m unittest` at the root level of the project to run all unit tests. # process_podcast This script enables you to assemble an ingterated podcast of a lecture or class from several different inputs, including: * recorded audio * screen recordings * other video * JPEG images representing individual lecture slides * a PDF containing the lecture slides Audio and video may come from the same file, or from separate files. You also have the ability to insert small “joiner” sections of audio and video between major “segments” of the podcast. You can even configure the “joiner” so that it repeats the last frame of the previous segment for the desired duration. For help on command line options: `process_podcast.py --help`. For help on the podcast configuration file format, see the [configuration file documentation][]. ## Requirements * Python 3.7 or later. * The `pyparsing` module. Linux: `pip install pyparsing`, `easy_install pyparsing`, or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-parsing`. Also see the [pyparsing documentation][]. * The `pexpect module`. Linux: `pip install pexpect`, `easy_install pexpect`, or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-pexpect`. Also see the [pexpect documentation][]. [pyparsing documentation]: http://pyparsing.wikispaces.com/ "pyparsing documentation" [pexpect documentation]: https://pexpect.readthedocs.io/en/stable/ "pexpect documentation" [configuration file documentation]: config_help.md "configuration file documentation" ## Testing Run `python -m unittest` at the root level of the project to run all unit tests.

Ignore Space Show notes View

README.md

# process_podcast

This script enables you to assemble an ingterated podcast of a lecture or class from several different inputs, including:

* recorded audio
* screen recordings
* other video
* JPEG images representing individual lecture slides
* a PDF containing the lecture slides

Audio and video may come from the same file, or from separate files. You also have the ability to insert small “joiner” sections of audio and video between major “segments” of the podcast. You can even configure the “joiner” so that it repeats the last frame of the previous segment for the desired duration.

For help on command line options: `process_podcast.py --help`.

For help on the podcast configuration file format, see the [configuration file documentation][].

## Requirements

* FFmpeg.
* ImageMagick.
* Python 3.5 or later with the following modules:
  * `pyparsing`; Linux: `pip install pyparsing`, `easy_install pyparsing`,  or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-parsing`. Also see the [pyparsing documentation][].
  * `pexpect`; Linux: `pip install pexpect`, `easy_install pexpect`,  or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-pexpect`. Also see the [pexpect documentation][].
  * There is a `requirements.txt` that you can use to quickly install all the required modules.

[pyparsing documentation]: http://pyparsing.wikispaces.com/ "pyparsing documentation"
[pexpect documentation]: https://pexpect.readthedocs.io/en/stable/ "pexpect documentation"
[configuration file documentation]: config_help.md "configuration file documentation"

## Testing

Run `python -m unittest` at the root level of the project to run all unit tests.

# process_podcast

This script enables you to assemble an ingterated podcast of a lecture or class from several different inputs, including:

* recorded audio
* screen recordings
* other video
* JPEG images representing individual lecture slides
* a PDF containing the lecture slides

For help on command line options: `process_podcast.py --help`.

For help on the podcast configuration file format, see the [configuration file documentation][].

## Requirements

* Python 3.7 or later.
* The `pyparsing` module. Linux: `pip install pyparsing`, `easy_install pyparsing`,  or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-parsing`. Also see the [pyparsing documentation][].
* The `pexpect module`.  Linux: `pip install pexpect`, `easy_install pexpect`,  or whatever other method you normally use to install Python modules; macOS (MacPorts): `port install py-pexpect`. Also see the [pexpect documentation][].

## Testing

Run `python -m unittest` at the root level of the project to run all unit tests.

Ignore Space Show notes View process_podcast.py
#!/usr/bin/env python3 import argparse import datetime import logging from pathlib import Path, PurePath import sys from pyparsing import ParseResults import globals from config_parser import ( parse_configuration_file, parse_configuration_string ) from progress_bar import ProgressBar from segment import ( Segment, AudioSegment, VideoSegment, FrameSegment, SegmentError ) from shell_command import FFprobeCommand, FFmpegConcatCommand class InputStreamAction(argparse.Action): def __call__(self, parser, namespace, values, option_string=None): input = values.split(":") file = input[0] stream = None if (len(input) == 1) else input[1] setattr(namespace, self.dest, file) if (option_string in ["--audio", "-a"]): setattr(namespace, 'audio_stream_number', stream) elif (option_string in ["--video", "-v"]): setattr(namespace, 'video_stream_number', stream) def parse_command_line(): """Parse command line arguments.""" parser = argparse.ArgumentParser( usage="%(prog)s [options] <output>", description="where: <output> is the name of the output file " "(note: .mov seems generally best)", epilog="Default input files can be specified using either of " "--audio or --video. If neither of these are specified, " "then you must supply a configuration file using --config. " "(Of course, you can always supply a configuration file " "regardless.)\n\n" "Input streams can be taken from the same input file.\n\n" "If no segments are specified, the entire input stream is " "processed as one segment. The number and duration of segments " "can differ, but the total duration across all input streams " "should ideally be the same.") parser.set_defaults(audio_stream_number=None, video_stream_number=None) parser.add_argument( "output", help="name of the output file (note: .mov is best)") parser.add_argument( "--audio", "-a", metavar="FILE[:STREAM]", action=InputStreamAction, help="File name for the default audio input stream (can be the " "same as for other input streams). You can optionally specify " "the default audio stream number to use if the file contains " "more than one (this can be overidden in a configuration " "file). If you don't specify a stream number, it defaults " "to 0 (i.e., the first audio stream in the file).") parser.add_argument( "--video", "-v", metavar="FILE[:STREAM]", action=InputStreamAction, help="File name for the default video input stream (can be the " "same as for other input streams). You can optionally specify " "the default video stream number to use if the file contains " "more than one (this can be overidden in a configuration " "file). If you don't specify a stream number, it defaults " "to 0 (i.e., the first video stream in the file).") parser.add_argument( "--configuration", "--config", "-c", dest="config", metavar="FILE", help="File name for the podcast segment configuration (plain text). " "See config_help.md details on the file " "format.".format(p=globals.PROGRAM)) parser.add_argument( "--copy-audio", dest="process_audio", action="store_false", default=True, help="Disable additional processing of the source audio. The audio " "will still be re-encoded using the specified audio codec " "because the concatenation filter requires it, but extra " "processing such as reduction of channels or normalisation " "will not be carried out. (Implies --no-normalise.)") parser.add_argument( "--copy-video", dest="process_video", action="store_false", default=True, help="Disable additional processing of the source video. The video " "will still be re-encoded using the specified video codec " "because the concatenation filter requires it, but extra " "processing such as remapping of colours will not be " "carried out.") parser.add_argument( "--no-normalise", dest="normalise", action="store_false", default=True, help="Disable normalisation of the source audio level (implied by " "--copy-audio).") parser.add_argument( "--audio-codec", dest="audio_codec", metavar="CODEC", default="pcm_s16le", help="Specify ffmpeg audio codec for output (default pcm_s16le). " "See the output of ffmpeg -codecs for possible codecs.") parser.add_argument( "--video-codec", dest="video_codec", metavar="CODEC", default="h264", help="Specify ffmpeg video codec for output (default h264). " "See the output of ffmpeg -codecs for possible codecs.") parser.add_argument( "--input-prefix", "-i", dest="prefix", metavar="PATH", default=".", help="Path to be prefixed to all INPUT files. This includes the " "configuration file, if applicable, and any files specified " "within the configuration file. Input files that already " "include the input prefix will not have it added again.") parser.add_argument( "--preview", "-p", metavar="RATE", nargs="?", const="1", help="Generate a preview of the podcast by only rendering a " "subset of the video frames. RATE is the number of frames " "per second to render (default 1 fps). You can specify " "fractions (e.g., 1/10 for one frame every 10 seconds). " 'NOTE: if you get a "too few arguments" error when using ' "this option, you've probably not provided an fps value and " "placed the option just before the output filename. " "Either move --preview earlier in the option list, add " 'a "--" between it and the output filename, or provide a ' "fps value.") parser.add_argument( "--debug", "-d", action="store_true", help="Print debugging output (overrides --quiet).") parser.add_argument( "--keep", "-k", action="store_true", help="Don't delete any generated temporary files.") parser.add_argument( "--quiet", "-q", action="store_true", help="Mute all console output (overridden by --debug).") args = parser.parse_args() return args def prefix_path(prefix, path): """Prepend prefix to path, unless path already starts with prefix""" assert(prefix is not None) if not path: return None elif path == "." or path.startswith(prefix): return path else: return Path(prefix, path) def check_arguments(args): """Sanity check the command line arguments.""" fn = "check_arguments" # Prepend input files with --input-prefix where applicable. args.audio, args.video, args.config = map( prefix_path, [args.prefix] * 3, [args.audio, args.video, args.config]) if args.quiet: globals.log.setLevel(logging.WARNING) # --copy-audio implies --no-normalise. if not args.process_audio: args.normalise = False # --debug overrides --quiet. if args.debug: globals.log.setLevel(logging.DEBUG) globals.log.debug("{fn}(): args = {a}".format(fn=fn, a=args)) # Must specify at least one of --audio, --video, --config. if not any([args.audio, args.video, args.config]): globals.log.error("must specify at least one of --audio, --video, " "or --config") sys.exit(1) if not Path(args.prefix).exists(): globals.log.error('input prefix "{p}" does not ' "exist".format(p=args.prefix)) sys.exit(1) def get_configuration(args): """Load podcast configuration.""" # Fill in missing file names for default input streams. fn = "get_configuration" # These types can have default input files and streams. type_mapping = { "audio": {"file": args.audio, "stream": args.audio_stream_number}, "video": {"file": args.video, "stream": args.video_stream_number}} globals.log.info("Processing configuration...") if (args.config): config = parse_configuration_file(args.config) # Check that applicable default input streams have been specified. for i, c in enumerate(config): type = c["type"] # Add prefix to filename, if applicable. if c["filename"] and (c["filename"] != "^"): config[i]["filename"] = prefix_path(args.prefix, config[i]["filename"]) if (type in type_mapping): default_file = type_mapping[type]["file"] default_stream = type_mapping[type]["stream"] error_string = ("attempting to use default {s} input file, " "but --{s} hasn't been specified".format(s=type)) else: default_file = None default_stream = 0 error_string = ("attempting to use a default input file, " "but the {s} type doesn't support this".format(s=type)) # No filename in configuration. if (not c["filename"]): if (default_file): config[i]["filename"] = default_file # No filename on command line either. else: globals.log.error(error_string) sys.exit(1) # No stream number in configuration. Note: 0 is a valid # stream number, so explicitly check for None. if (c["num"] is None): # Assume 0 if no stream on command line either. if (default_stream is None): config[i]["num"] = 0 else: config[i]["num"] = default_stream # No configuration file. else: conf_list = [] for m in type_mapping: default_file = type_mapping[m]["file"] default_stream = type_mapping[m]["stream"] if (default_file and default_stream is not None): conf_list += ["[{type}:{file}:{stream}]".format( type=m, file=default_file, stream=default_stream)] globals.log.debug("{fn}(): default config = " "{c}".format(fn=fn, c=conf_list)) config = parse_configuration_string("\n".join(conf_list)) return config def get_file_duration(file): """Calculate the duration a media file as a timedelta object.""" fn = "get_file_duration" command = FFprobeCommand([file]) globals.log.debug("{fn}(): {cmd}".format(fn=fn, cmd=command)) # Only consider the first stream. If it's the only stream in the # file, great; otherwise it seems reasonable to assume that all # streams in the same file will have the same duration. ss, ms = command.get_entries( section="format", find_list=["duration"])[0].split(".") ms = ms[:3].ljust(3, "0") globals.log.debug("{fn}(): ss = {ss}, ms = {ms}".format(fn=fn, ss=ss, ms=ms)) return datetime.timedelta(seconds=int(ss), milliseconds=int(ms)) def make_new_segment(type, filename, punch_in, punch_out, num): """Make a new segment instance of the correct class.""" fn = "make_new_segment" globals.log.debug("{fn}(): type = {t}".format(fn=fn, t=type)) globals.log.debug("{fn}(): filename = {f}".format(fn=fn, f=filename)) globals.log.debug("{fn}(): punch in = {i}".format(fn=fn, i=punch_in)) globals.log.debug("{fn}(): punch out = {o}".format(fn=fn, o=punch_out)) globals.log.debug("{fn}(): num = {n}".format(fn=fn, n=num)) if (type == "audio"): return AudioSegment(file=filename, punch_in=punch_in, punch_out=punch_out, input_stream=num) elif (type == "video"): return VideoSegment(file=filename, punch_in=punch_in, punch_out=punch_out, input_stream=num) elif (type == "frame"): return FrameSegment(file=filename, punch_in=punch_in, punch_out=punch_out, frame_number=num) else: return None def process_timestamp_pair(args, times): """Constructs timedelta instances from a pair of config timestamps.""" fn = "process_timestamp_pair" globals.log.debug("{fn}(): times[0] = {t}".format(fn=fn, t=times[0])) globals.log.debug("{fn}(): times[1] = {t}".format(fn=fn, t=times[1])) # If the first item in the timestamp list in the configuration file # is a filename, the parser inserts a zero timestamp before it. We # can therefore guarantee that the first item of the pair will # always be a timestamp. t0 = datetime.timedelta( hours=times[0]["hh"], minutes=times[0]["mm"], seconds=times[0]["ss"], milliseconds=times[0]["ms"]) if (times[1]): if (len(times[1]) == 1): # filename t1 = t0 + get_file_duration( prefix_path(args.prefix, times[1]["filename"])) elif (len(times[1]) == 4): # normal timestamp t1 = datetime.timedelta( hours=times[1]["hh"], minutes=times[1]["mm"], seconds=times[1]["ss"], milliseconds=times[1]["ms"]) else: globals.log.error("{fn}():unreadable timestamp {t}".format( fn=fn, t=times[1])) t1 = None else: t1 = None globals.log.debug("{fn}(): t0 = {t}".format(fn=fn, t=t0)) globals.log.debug("{fn}(): t1 = {t}".format(fn=fn, t=t1)) return t0, t1 def process_time_list(args, type, filename, num, time_list): """Process an audio or video stream and build a list of segments.""" fn = "process_time_list" if (Path(filename).exists() and type in ["audio", "video"]): stream_duration = get_file_duration(filename) else: stream_duration = 0 segments = [] globals.log.debug("{fn}(): stream duration = {d}".format( fn=fn, d=stream_duration)) # No timestamps: punch in at 0, out at stream duration. if (len(time_list) == 0): punch_in = datetime.timedelta() punch_out = stream_duration segments.append(make_new_segment(type, filename, punch_in, punch_out, num)) else: # Process each pair of timestamps as punch in, out. If there's # an odd number of items, the last one is processed separately. for t in zip(time_list[::2], time_list[1::2]): punch_in, punch_out = process_timestamp_pair(args, t) if (punch_in == punch_out): globals.log.warning( "punch in ({i}s) and punch out ({o}s) times are " "equal; no segment will be " "generated".format(i=punch_in.total_seconds(), o=punch_out.total_seconds())) continue elif (punch_out < punch_in): globals.log.error( "punch out time ({i}s) falls before punch in time " "({o}s); can't generate a valid " "segment".format(i=punch_in.total_seconds(), o=punch_out.total_seconds())) sys.exit(1) segments.append(make_new_segment(type, filename, punch_in, punch_out, num)) # Odd number of timestamps: punch in at last timestamp, # out at stream duration. if (len(time_list) % 2 != 0): globals.log.debug("{fn}(): odd number of timestamps".format(fn=fn)) punch_in, _ = process_timestamp_pair(args, [time_list[-1], None]) punch_out = stream_duration segments.append(make_new_segment(type, filename, punch_in, punch_out, num)) return segments def process_input_streams(args, config): """Process a list of stream specifications and build a list of segments.""" fn = "process_input_streams" globals.log.info("Processing input streams...") segments = [] for cnf in config: globals.log.debug("{fn}(): type = {t}".format(fn=fn, t=cnf["type"])) globals.log.debug( "{fn}(): filename = {f}".format(fn=fn, f=cnf["filename"])) globals.log.debug("{fn}(): num = {n}".format(fn=fn, n=cnf["num"])) globals.log.debug("{fn}(): times = {t}".format(fn=fn, t=cnf["times"])) segments += process_time_list(args, cnf["type"], cnf["filename"], cnf["num"], cnf["times"]) return segments def smallest_video_dimensions(args, segments): """Compute the smallest frame dimensions across all video inputs.""" fn = "smallest_video_dimensions" width = 2048 height = 1536 for s in segments: if isinstance(s, FrameSegment): continue command = FFprobeCommand([s.input_file]) globals.log.debug("{fn}(): {cmd}".format(fn=fn, cmd=command)) w, h = command.get_entries(find_list=["width", "height"]) if (w * h) < (width * height): width, height = w, h return width, height def process_frame_segments(args, segments, width, height): """Post-process frame segments to set frame images, etc.""" fn = "process_frame_segments" globals.log.info("Processing frames...") frame_segments = [s for s in segments if isinstance(s, FrameSegment)] n = len(frame_segments) globals.log.debug("{fn}(): num frames = {n}".format(fn=fn, n=n)) progress = ProgressBar(max_value=n, quiet=args.quiet or args.debug or n == 0) progress.update(0) for i, f in enumerate(frame_segments): try: globals.log.debug( "{fn}(): frame (before) = {b}".format(fn=fn, b=f)) # Frame segments that use a frame from the previous segment. if (f.input_file == "^"): if (f.segment_number > 0): prev = segments[f.segment_number - 1] globals.log.debug( "{fn}(): prev = {p}".format(fn=fn, p=prev)) prev.generate_temp_file(args.output, width=width, height=height) f.use_frame( prev.generate_frame(f.frame_number, args.output, width=width, height=height)) else: globals.log.error( "frame segment {s} is attempting to use the last " "frame of a non-existent previous " "segment".format(s=f.segment_number)) sys.exit(1) # Frame segments whose frame comes from a PDF file. else: suffix = PurePath(f.input_file).suffix if (suffix.lower() == ".pdf"): f.use_frame(f.generate_temp_file(args.output, width=width, height=height)) else: globals.log.error( 'unexpected input file type "{s}" for frame segment ' "{f}".format(s=suffix, f=f.segment_number)) sys.exit(1) progress.update(i) globals.log.debug("{fn}(): frame (after) = ""{a}".format(fn=fn, a=f)) except SegmentError as e: progress.finish() globals.log.exception(e) sys.exit(1) else: progress.finish() def render_podcast(args, audio_segments, video_segments, output, duration): """Stitch together the various input components into the final podcast.""" fn = "render_podcast" globals.log.info("Rendering final podcast...") command = FFmpegConcatCommand(has_audio=len(audio_segments) > 0, has_video=len(video_segments) > 0, max_progress=duration, quiet=args.quiet and not args.debug, process_audio=args.process_audio, process_video=args.process_video, audio_codec=args.audio_codec, video_codec=args.video_codec) input_files = Segment.input_files() for f in input_files: if (input_files[f]): command.append_input_options(input_files[f]) command.append_input_options(["-i", f]) for s in (audio_segments + video_segments): command.append_filter(s.trim_filter()) command.append_concat_filter("a", [s for s in audio_segments]) if (args.normalise): command.append_normalisation_filter() command.append_concat_filter("v", [s for s in video_segments]) if args.preview: globals.log.info("PREVIEW MODE: {fps} fps".format(fps=args.preview)) command.append_output_options(["-r", args.preview]) command.append_output_options([output]) globals.log.debug("{fn}(): {c}".format(fn=fn, c=command)) if (command.run() != 0): globals.log.error("Failed to render final podcast") def cleanup(segments): """Clean up generated temporary files.""" globals.log.info("Cleaning up...") for s in segments: s.delete_temp_files() def main(): fn = "main" logging.basicConfig( level=logging.INFO, format="%(levelname)s: {p}: %(message)s".format(p=globals.PROGRAM)) segments = None try: args = parse_command_line() check_arguments(args) config = get_configuration(args) segments = process_input_streams(args, config) globals.log.debug("{fn}(): audio segments = {a}".format( fn=fn, a=[s for s in segments if isinstance(s, AudioSegment)])) globals.log.debug("{fn}(): video segments = {v}".format( fn=fn, v=[s for s in segments if isinstance(s, VideoSegment)])) audio_segments = [s for s in segments if isinstance(s, AudioSegment)] video_segments = [s for s in segments if isinstance(s, VideoSegment)] audio_duration = sum([s.get_duration() for s in audio_segments]) video_duration = sum([s.get_duration() for s in video_segments]) globals.log.debug("{fn}(): audio duration = " "{a}".format(fn=fn, a=audio_duration)) globals.log.debug("{fn}(): video duration = " "{v}".format(fn=fn, v=video_duration)) if (len(audio_segments) and len(video_segments)): if (audio_duration != video_duration): globals.log.warning("total video duration ({v}s) doesn't match " "total audio duration " "({a}s)".format(v=video_duration, a=audio_duration)) width, height = smallest_video_dimensions(args, video_segments) globals.log.debug("{fn}(): width = {w}, height = " "{h}".format(fn=fn, w=width, h=height)) process_frame_segments(args, segments, width, height) globals.log.debug("{fn}(): input files = " "{i}".format(fn=fn, i=Segment.input_files())) render_podcast(args, audio_segments, video_segments, args.output, max(audio_duration, video_duration)) except (KeyboardInterrupt): pass finally: if segments and not args.keep: cleanup(segments) if (__name__ == "__main__"): assert sys.version_info >= (3, 5) main() #!/usr/bin/env python3 import argparse import datetime import logging from pathlib import Path, PurePath import sys from pyparsing import ParseResults import globals from config_parser import ( parse_configuration_file, parse_configuration_string ) from progress_bar import ProgressBar from segment import ( Segment, AudioSegment, VideoSegment, FrameSegment, SegmentError ) from shell_command import FFprobeCommand, FFmpegConcatCommand class InputStreamAction(argparse.Action): def __call__(self, parser, namespace, values, option_string=None): input = values.split(":") file = input[0] stream = None if (len(input) == 1) else input[1] setattr(namespace, self.dest, file) if (option_string in ["--audio", "-a"]): setattr(namespace, 'audio_stream_number', stream) elif (option_string in ["--video", "-v"]): setattr(namespace, 'video_stream_number', stream) def parse_command_line(): """Parse command line arguments.""" parser = argparse.ArgumentParser( usage="%(prog)s [options] <output>", description="where: <output> is the name of the output file " "(note: .mov seems generally best)", epilog="Default input files can be specified using either of " "--audio or --video. If neither of these are specified, " "then you must supply a configuration file using --config. " "(Of course, you can always supply a configuration file " "regardless.)\n\n" "Input streams can be taken from the same input file.\n\n" "If no segments are specified, the entire input stream is " "processed as one segment. The number and duration of segments " "can differ, but the total duration across all input streams " "should ideally be the same.") parser.set_defaults(audio_stream_number=None, video_stream_number=None) parser.add_argument( "output", help="name of the output file (note: .mov is best)") parser.add_argument( "--audio", "-a", metavar="FILE[:STREAM]", action=InputStreamAction, help="File name for the default audio input stream (can be the " "same as for other input streams). You can optionally specify " "the default audio stream number to use if the file contains " "more than one (this can be overidden in a configuration " "file). If you don't specify a stream number, it defaults " "to 0 (i.e., the first audio stream in the file).") parser.add_argument( "--video", "-v", metavar="FILE[:STREAM]", action=InputStreamAction, help="File name for the default video input stream (can be the " "same as for other input streams). You can optionally specify " "the default video stream number to use if the file contains " "more than one (this can be overidden in a configuration " "file). If you don't specify a stream number, it defaults " "to 0 (i.e., the first video stream in the file).") parser.add_argument( "--configuration", "--config", "-c", dest="config", metavar="FILE", help="File name for the podcast segment configuration (plain text). " "See config_help.md details on the file " "format.".format(p=globals.PROGRAM)) parser.add_argument( "--copy-audio", dest="process_audio", action="store_false", default=True, help="Disable additional processing of the source audio. The audio " "will still be re-encoded using the specified audio codec " "because the concatenation filter requires it, but extra " "processing such as reduction of channels or normalisation " "will not be carried out. (Implies --no-normalise.)") parser.add_argument( "--copy-video", dest="process_video", action="store_false", default=True, help="Disable additional processing of the source video. The video " "will still be re-encoded using the specified video codec " "because the concatenation filter requires it, but extra " "processing such as remapping of colours will not be " "carried out.") parser.add_argument( "--no-normalise", dest="normalise", action="store_false", default=True, help="Disable normalisation of the source audio level (implied by " "--copy-audio).") parser.add_argument( "--audio-codec", dest="audio_codec", metavar="CODEC", default="pcm_s16le", help="Specify ffmpeg audio codec for output (default pcm_s16le). " "See the output of ffmpeg -codecs for possible codecs.") parser.add_argument( "--video-codec", dest="video_codec", metavar="CODEC", default="h264", help="Specify ffmpeg video codec for output (default h264). " "See the output of ffmpeg -codecs for possible codecs.") parser.add_argument( "--input-prefix", "-i", dest="prefix", metavar="PATH", default=".", help="Path to be prefixed to all INPUT files. This includes the " "configuration file, if applicable, and any files specified " "within the configuration file. Input files that already " "include the input prefix will not have it added again.") parser.add_argument( "--preview", "-p", metavar="RATE", nargs="?", const="1", help="Generate a preview of the podcast by only rendering a " "subset of the video frames. RATE is the number of frames " "per second to render (default 1 fps). You can specify " "fractions (e.g., 1/10 for one frame every 10 seconds). " 'NOTE: if you get a "too few arguments" error when using ' "this option, you've probably not provided an fps value and " "placed the option just before the output filename. " "Either move --preview earlier in the option list, add " 'a "--" between it and the output filename, or provide a ' "fps value.") parser.add_argument( "--debug", "-d", action="store_true", help="Print debugging output (overrides --quiet).") parser.add_argument( "--keep", "-k", action="store_true", help="Don't delete any generated temporary files.") parser.add_argument( "--quiet", "-q", action="store_true", help="Mute all console output (overridden by --debug).") args = parser.parse_args() return args def prefix_path(prefix, path): """Prepend prefix to path, unless path already starts with prefix""" assert(prefix is not None) if not path: return None elif path == "." or path.startswith(prefix): return path else: return Path(prefix, path) def check_arguments(args): """Sanity check the command line arguments.""" fn = "check_arguments" # Prepend input files with --input-prefix where applicable. args.audio, args.video, args.config = map( prefix_path, [args.prefix] * 3, [args.audio, args.video, args.config]) if args.quiet: globals.log.setLevel(logging.WARNING) # --copy-audio implies --no-normalise. if not args.process_audio: args.normalise = False # --debug overrides --quiet. if args.debug: globals.log.setLevel(logging.DEBUG) globals.log.debug("{fn}(): args = {a}".format(fn=fn, a=args)) # Must specify at least one of --audio, --video, --config. if not any([args.audio, args.video, args.config]): globals.log.error("must specify at least one of --audio, --video, " "or --config") sys.exit(1) if not Path(args.prefix).exists(): globals.log.error('input prefix "{p}" does not ' "exist".format(p=args.prefix)) sys.exit(1) def get_configuration(args): """Load podcast configuration.""" # Fill in missing file names for default input streams. fn = "get_configuration" # These types can have default input files and streams. type_mapping = { "audio": {"file": args.audio, "stream": args.audio_stream_number}, "video": {"file": args.video, "stream": args.video_stream_number}} globals.log.info("Processing configuration...") if (args.config): config = parse_configuration_file(args.config) # Check that applicable default input streams have been specified. for i, c in enumerate(config): type = c["type"] # Add prefix to filename, if applicable. if c["filename"] and (c["filename"] != "^"): config[i]["filename"] = prefix_path(args.prefix, config[i]["filename"]) if (type in type_mapping): default_file = type_mapping[type]["file"] default_stream = type_mapping[type]["stream"] error_string = ("attempting to use default {s} input file, " "but --{s} hasn't been specified".format(s=type)) else: default_file = None default_stream = 0 error_string = ("attempting to use a default input file, " "but the {s} type doesn't support this".format(s=type)) # No filename in configuration. if (not c["filename"]): if (default_file): config[i]["filename"] = default_file # No filename on command line either. else: globals.log.error(error_string) sys.exit(1) # No stream number in configuration. Note: 0 is a valid # stream number, so explicitly check for None. if (c["num"] is None): # Assume 0 if no stream on command line either. if (default_stream is None): config[i]["num"] = 0 else: config[i]["num"] = default_stream # No configuration file. else: conf_list = [] for m in type_mapping: default_file = type_mapping[m]["file"] default_stream = type_mapping[m]["stream"] if (default_file and default_stream is not None): conf_list += ["[{type}:{file}:{stream}]".format( type=m, file=default_file, stream=default_stream)] globals.log.debug("{fn}(): default config = " "{c}".format(fn=fn, c=conf_list)) config = parse_configuration_string("\n".join(conf_list)) return config def get_file_duration(file): """Calculate the duration a media file as a timedelta object.""" fn = "get_file_duration" command = FFprobeCommand([file]) globals.log.debug("{fn}(): {cmd}".format(fn=fn, cmd=command)) # Only consider the first stream. If it's the only stream in the # file, great; otherwise it seems reasonable to assume that all # streams in the same file will have the same duration. ss, ms = command.get_entries( section="format", find_list=["duration"])[0].split(".") ms = ms[:3].ljust(3, "0") globals.log.debug("{fn}(): ss = {ss}, ms = {ms}".format(fn=fn, ss=ss, ms=ms)) return datetime.timedelta(seconds=int(ss), milliseconds=int(ms)) def make_new_segment(type, filename, punch_in, punch_out, num): """Make a new segment instance of the correct class.""" fn = "make_new_segment" globals.log.debug("{fn}(): type = {t}".format(fn=fn, t=type)) globals.log.debug("{fn}(): filename = {f}".format(fn=fn, f=filename)) globals.log.debug("{fn}(): punch in = {i}".format(fn=fn, i=punch_in)) globals.log.debug("{fn}(): punch out = {o}".format(fn=fn, o=punch_out)) globals.log.debug("{fn}(): num = {n}".format(fn=fn, n=num)) if (type == "audio"): return AudioSegment(file=filename, punch_in=punch_in, punch_out=punch_out, input_stream=num) elif (type == "video"): return VideoSegment(file=filename, punch_in=punch_in, punch_out=punch_out, input_stream=num) elif (type == "frame"): return FrameSegment(file=filename, punch_in=punch_in, punch_out=punch_out, frame_number=num) else: return None def process_timestamp_pair(args, times): """Constructs timedelta instances from a pair of config timestamps.""" fn = "process_timestamp_pair" globals.log.debug("{fn}(): times[0] = {t}".format(fn=fn, t=times[0])) globals.log.debug("{fn}(): times[1] = {t}".format(fn=fn, t=times[1])) # If the first item in the timestamp list in the configuration file # is a filename, the parser inserts a zero timestamp before it. We # can therefore guarantee that the first item of the pair will # always be a timestamp. t0 = datetime.timedelta( hours=times[0]["hh"], minutes=times[0]["mm"], seconds=times[0]["ss"], milliseconds=times[0]["ms"]) if (times[1]): if (len(times[1]) == 1): # filename t1 = t0 + get_file_duration( prefix_path(args.prefix, times[1]["filename"])) elif (len(times[1]) == 4): # normal timestamp t1 = datetime.timedelta( hours=times[1]["hh"], minutes=times[1]["mm"], seconds=times[1]["ss"], milliseconds=times[1]["ms"]) else: globals.log.error("{fn}():unreadable timestamp {t}".format( fn=fn, t=times[1])) t1 = None else: t1 = None globals.log.debug("{fn}(): t0 = {t}".format(fn=fn, t=t0)) globals.log.debug("{fn}(): t1 = {t}".format(fn=fn, t=t1)) return t0, t1 def process_time_list(args, type, filename, num, time_list): """Process an audio or video stream and build a list of segments.""" fn = "process_time_list" if (Path(filename).exists() and type in ["audio", "video"]): stream_duration = get_file_duration(filename) else: stream_duration = 0 segments = [] globals.log.debug("{fn}(): stream duration = {d}".format( fn=fn, d=stream_duration)) # No timestamps: punch in at 0, out at stream duration. if (len(time_list) == 0): punch_in = datetime.timedelta() punch_out = stream_duration segments.append(make_new_segment(type, filename, punch_in, punch_out, num)) else: # Process each pair of timestamps as punch in, out. If there's # an odd number of items, the last one is processed separately. for t in zip(time_list[::2], time_list[1::2]): punch_in, punch_out = process_timestamp_pair(args, t) if (punch_in == punch_out): globals.log.warning( "punch in ({i}s) and punch out ({o}s) times are " "equal; no segment will be " "generated".format(i=punch_in.total_seconds(), o=punch_out.total_seconds())) continue elif (punch_out < punch_in): globals.log.error( "punch out time ({i}s) falls before punch in time " "({o}s); can't generate a valid " "segment".format(i=punch_in.total_seconds(), o=punch_out.total_seconds())) sys.exit(1) segments.append(make_new_segment(type, filename, punch_in, punch_out, num)) # Odd number of timestamps: punch in at last timestamp, # out at stream duration. if (len(time_list) % 2 != 0): globals.log.debug("{fn}(): odd number of timestamps".format(fn=fn)) punch_in, _ = process_timestamp_pair(args, [time_list[-1], None]) punch_out = stream_duration segments.append(make_new_segment(type, filename, punch_in, punch_out, num)) return segments def process_input_streams(args, config): """Process a list of stream specifications and build a list of segments.""" fn = "process_input_streams" globals.log.info("Processing input streams...") segments = [] for cnf in config: globals.log.debug("{fn}(): type = {t}".format(fn=fn, t=cnf["type"])) globals.log.debug( "{fn}(): filename = {f}".format(fn=fn, f=cnf["filename"])) globals.log.debug("{fn}(): num = {n}".format(fn=fn, n=cnf["num"])) globals.log.debug("{fn}(): times = {t}".format(fn=fn, t=cnf["times"])) segments += process_time_list(args, cnf["type"], cnf["filename"], cnf["num"], cnf["times"]) return segments def smallest_video_dimensions(args, segments): """Compute the smallest frame dimensions across all video inputs.""" fn = "smallest_video_dimensions" width = 2048 height = 1536 for s in segments: if isinstance(s, FrameSegment): continue command = FFprobeCommand([s.input_file]) globals.log.debug("{fn}(): {cmd}".format(fn=fn, cmd=command)) w, h = command.get_entries(find_list=["width", "height"]) if (w * h) < (width * height): width, height = w, h return width, height def process_frame_segments(args, segments, width, height): """Post-process frame segments to set frame images, etc.""" fn = "process_frame_segments" globals.log.info("Processing frames...") frame_segments = [s for s in segments if isinstance(s, FrameSegment)] n = len(frame_segments) globals.log.debug("{fn}(): num frames = {n}".format(fn=fn, n=n)) progress = ProgressBar(max_value=n, quiet=args.quiet or args.debug or n == 0) progress.update(0) for i, f in enumerate(frame_segments): try: globals.log.debug( "{fn}(): frame (before) = {b}".format(fn=fn, b=f)) # Frame segments that use a frame from the previous segment. if (f.input_file == "^"): if (f.segment_number > 0): prev = segments[f.segment_number - 1] globals.log.debug( "{fn}(): prev = {p}".format(fn=fn, p=prev)) prev.generate_temp_file(args.output, width=width, height=height) f.use_frame( prev.generate_frame(f.frame_number, args.output, width=width, height=height)) else: globals.log.error( "frame segment {s} is attempting to use the last " "frame of a non-existent previous " "segment".format(s=f.segment_number)) sys.exit(1) # Frame segments whose frame comes from a PDF file. else: suffix = PurePath(f.input_file).suffix if (suffix.lower() == ".pdf"): f.use_frame(f.generate_temp_file(args.output, width=width, height=height)) else: globals.log.error( 'unexpected input file type "{s}" for frame segment ' "{f}".format(s=suffix, f=f.segment_number)) sys.exit(1) progress.update(i) globals.log.debug("{fn}(): frame (after) = ""{a}".format(fn=fn, a=f)) except SegmentError as e: progress.finish() globals.log.exception(e) sys.exit(1) else: progress.finish() def render_podcast(args, audio_segments, video_segments, output, duration): """Stitch together the various input components into the final podcast.""" fn = "render_podcast" globals.log.info("Rendering final podcast...") command = FFmpegConcatCommand(has_audio=len(audio_segments) > 0, has_video=len(video_segments) > 0, max_progress=duration, quiet=args.quiet and not args.debug, process_audio=args.process_audio, process_video=args.process_video, audio_codec=args.audio_codec, video_codec=args.video_codec) input_files = Segment.input_files() for f in input_files: if (input_files[f]): command.append_input_options(input_files[f]) command.append_input_options(["-i", f]) for s in (audio_segments + video_segments): command.append_filter(s.trim_filter()) command.append_concat_filter("a", [s for s in audio_segments]) if (args.normalise): command.append_normalisation_filter() command.append_concat_filter("v", [s for s in video_segments]) if args.preview: globals.log.info("PREVIEW MODE: {fps} fps".format(fps=args.preview)) command.append_output_options(["-r", args.preview]) command.append_output_options([output]) globals.log.debug("{fn}(): {c}".format(fn=fn, c=command)) if (command.run() != 0): globals.log.error("Failed to render final podcast") def cleanup(segments): """Clean up generated temporary files.""" globals.log.info("Cleaning up...") for s in segments: s.delete_temp_files() def main(): fn = "main" logging.basicConfig( level=logging.INFO, format="%(levelname)s: {p}: %(message)s".format(p=globals.PROGRAM)) segments = None try: args = parse_command_line() check_arguments(args) config = get_configuration(args) segments = process_input_streams(args, config) globals.log.debug("{fn}(): audio segments = {a}".format( fn=fn, a=[s for s in segments if isinstance(s, AudioSegment)])) globals.log.debug("{fn}(): video segments = {v}".format( fn=fn, v=[s for s in segments if isinstance(s, VideoSegment)])) audio_segments = [s for s in segments if isinstance(s, AudioSegment)] video_segments = [s for s in segments if isinstance(s, VideoSegment)] audio_duration = sum([s.get_duration() for s in audio_segments]) video_duration = sum([s.get_duration() for s in video_segments]) globals.log.debug("{fn}(): audio duration = " "{a}".format(fn=fn, a=audio_duration)) globals.log.debug("{fn}(): video duration = " "{v}".format(fn=fn, v=video_duration)) if (len(audio_segments) and len(video_segments)): if (audio_duration != video_duration): globals.log.warning("total video duration ({v}s) doesn't match " "total audio duration " "({a}s)".format(v=video_duration, a=audio_duration)) width, height = smallest_video_dimensions(args, video_segments) globals.log.debug("{fn}(): width = {w}, height = " "{h}".format(fn=fn, w=width, h=height)) process_frame_segments(args, segments, width, height) globals.log.debug("{fn}(): input files = " "{i}".format(fn=fn, i=Segment.input_files())) render_podcast(args, audio_segments, video_segments, args.output, max(audio_duration, video_duration)) except (KeyboardInterrupt): pass finally: if segments and not args.keep: cleanup(segments) if (__name__ == "__main__"): assert sys.version_info >= (3, 7) main()

Ignore Space Show notes View

process_podcast.py

#!/usr/bin/env python3

import argparse
import datetime
import logging
from pathlib import Path, PurePath
import sys

from pyparsing import ParseResults

import globals
from config_parser import (
    parse_configuration_file, parse_configuration_string
)
from progress_bar import ProgressBar
from segment import (
    Segment, AudioSegment, VideoSegment,
    FrameSegment, SegmentError
)
from shell_command import FFprobeCommand, FFmpegConcatCommand

class InputStreamAction(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        input = values.split(":")
        file = input[0]
        stream = None if (len(input) == 1) else input[1]
        setattr(namespace, self.dest, file)
        if (option_string in ["--audio", "-a"]):
            setattr(namespace, 'audio_stream_number', stream)
        elif (option_string in ["--video", "-v"]):
            setattr(namespace, 'video_stream_number', stream)

def parse_command_line():
    """Parse command line arguments."""
    parser = argparse.ArgumentParser(
        usage="%(prog)s [options] <output>",
        description="where: <output> is the name of the output file "
            "(note: .mov seems generally best)",
        epilog="Default input files can be specified using either of "
            "--audio or --video. If neither of these are specified, "
            "then you must supply a configuration file using --config. "
            "(Of course, you can always supply a configuration file "
            "regardless.)\n\n"
            "Input streams can be taken from the same input file.\n\n"
            "If no segments are specified, the entire input stream is "
            "processed as one segment. The number and duration of segments "
            "can differ, but the total duration across all input streams "
            "should ideally be the same.")
    
    parser.set_defaults(audio_stream_number=None, video_stream_number=None)

parser.add_argument(
        "output",
        help="name of the output file (note: .mov is best)")
    
    parser.add_argument(
        "--audio", "-a", metavar="FILE[:STREAM]", action=InputStreamAction,
        help="File name for the default audio input stream (can be the "
            "same as for other input streams). You can optionally specify "
            "the default audio stream number to use if the file contains "
            "more than one (this can be overidden in a configuration "
            "file). If you don't specify a stream number, it defaults "
            "to 0 (i.e., the first audio stream in the file).")
    
    parser.add_argument(
        "--video", "-v", metavar="FILE[:STREAM]", action=InputStreamAction,
        help="File name for the default video input stream (can be the "
            "same as for other input streams). You can optionally specify "
            "the default video stream number to use if the file contains "
            "more than one (this can be overidden in a configuration "
            "file). If you don't specify a stream number, it defaults "
            "to 0 (i.e., the first video stream in the file).")
    
    parser.add_argument(
        "--configuration", "--config", "-c", dest="config", metavar="FILE",
        help="File name for the podcast segment configuration (plain text). "
            "See config_help.md details on the file "
            "format.".format(p=globals.PROGRAM))
    
    parser.add_argument(
        "--copy-audio", dest="process_audio", action="store_false",
        default=True,
        help="Disable additional processing of the source audio. The audio "
            "will still be re-encoded using the specified audio codec "
            "because the concatenation filter requires it, but extra "
            "processing such as reduction of channels or normalisation "
            "will not be carried out. (Implies --no-normalise.)")
    
    parser.add_argument(
        "--copy-video", dest="process_video", action="store_false",
        default=True,
        help="Disable additional processing of the source video. The video "
            "will still be re-encoded using the specified video codec "
            "because the concatenation filter requires it, but extra "
            "processing such as remapping of colours will not be "
            "carried out.")
    
    parser.add_argument(
        "--no-normalise", dest="normalise", action="store_false",
        default=True,
        help="Disable normalisation of the source audio level (implied by "
            "--copy-audio).")
    
    parser.add_argument(
        "--audio-codec", dest="audio_codec", metavar="CODEC",
        default="pcm_s16le",
        help="Specify ffmpeg audio codec for output (default pcm_s16le). "
            "See the output of ffmpeg -codecs for possible codecs.")
    
    parser.add_argument(
        "--video-codec", dest="video_codec", metavar="CODEC", default="h264",
        help="Specify ffmpeg video codec for output (default h264). "
            "See the output of ffmpeg -codecs for possible codecs.")
    
    parser.add_argument(
        "--input-prefix", "-i", dest="prefix", metavar="PATH", default=".",
        help="Path to be prefixed to all INPUT files. This includes the "
            "configuration file, if applicable, and any files specified "
            "within the configuration file. Input files that already "
            "include the input prefix will not have it added again.")
    
    parser.add_argument(
        "--preview", "-p", metavar="RATE", nargs="?", const="1",
        help="Generate a preview of the podcast by only rendering a "
            "subset of the video frames. RATE is the number of frames "
            "per second to render (default 1 fps). You can specify "
            "fractions (e.g., 1/10 for one frame every 10 seconds). "
            'NOTE: if you get a "too few arguments" error when using '
            "this option, you've probably not provided an fps value and "
            "placed the option just before the output filename. "
            "Either move --preview earlier in the option list, add "
            'a "--" between it and the output filename, or provide a '
            "fps value.")
    
    parser.add_argument(
        "--debug", "-d", action="store_true",
        help="Print debugging output (overrides --quiet).")
    
    parser.add_argument(
        "--keep", "-k", action="store_true",
        help="Don't delete any generated temporary files.")
    
    parser.add_argument(
        "--quiet", "-q", action="store_true",
        help="Mute all console output (overridden by --debug).")

args = parser.parse_args()
    
    return args

def prefix_path(prefix, path):
    """Prepend prefix to path, unless path already starts with prefix"""
    assert(prefix is not None)
    if not path:
        return None
    elif path == "." or path.startswith(prefix):
        return path
    else:
        return Path(prefix, path)

def check_arguments(args):
    """Sanity check the command line arguments."""
    fn = "check_arguments"
    # Prepend input files with --input-prefix where applicable.
    args.audio, args.video, args.config = map(
        prefix_path, [args.prefix] * 3, [args.audio, args.video, args.config])

if args.quiet:
        globals.log.setLevel(logging.WARNING)
    
    # --copy-audio implies --no-normalise.
    if not args.process_audio:
        args.normalise = False
        
    # --debug overrides --quiet.
    if args.debug:
        globals.log.setLevel(logging.DEBUG)
        globals.log.debug("{fn}(): args = {a}".format(fn=fn, a=args))
    
    # Must specify at least one of --audio, --video, --config.
    if not any([args.audio, args.video, args.config]):
        globals.log.error("must specify at least one of --audio, --video, "
                          "or --config")
        sys.exit(1)
    
    if not Path(args.prefix).exists():
        globals.log.error('input prefix "{p}" does not '
                          "exist".format(p=args.prefix))
        sys.exit(1)

def get_configuration(args):
    """Load podcast configuration."""
    # Fill in missing file names for default input streams.
    fn = "get_configuration"
    # These types can have default input files and streams.
    type_mapping = {
        "audio": {"file": args.audio, "stream": args.audio_stream_number},
        "video": {"file": args.video, "stream": args.video_stream_number}}
    globals.log.info("Processing configuration...")
    if (args.config):
        config = parse_configuration_file(args.config)
        
        # Check that applicable default input streams have been specified.
        for i, c in enumerate(config):
            type = c["type"]
            
            # Add prefix to filename, if applicable.
            if c["filename"] and (c["filename"] != "^"):
                config[i]["filename"] = prefix_path(args.prefix,
                                                    config[i]["filename"])
            
            if (type in type_mapping):
                default_file = type_mapping[type]["file"]
                default_stream = type_mapping[type]["stream"]
                error_string = ("attempting to use default {s} input file, "
                    "but --{s} hasn't been specified".format(s=type))
            else:
                default_file = None
                default_stream = 0
                error_string = ("attempting to use a default input file, "
                    "but the {s} type doesn't support this".format(s=type))

# No filename in configuration.
            if (not c["filename"]):
                if (default_file):
                    config[i]["filename"] = default_file
                # No filename on command line either.
                else:
                    globals.log.error(error_string)
                    sys.exit(1)
            
            # No stream number in configuration. Note: 0 is a valid
            # stream number, so explicitly check for None.
            if (c["num"] is None):
                # Assume 0 if no stream on command line either.
                if (default_stream is None):
                    config[i]["num"] = 0
                else:
                    config[i]["num"] = default_stream
    
    # No configuration file.
    else:
        conf_list = []
        for m in type_mapping:
            default_file = type_mapping[m]["file"]
            default_stream = type_mapping[m]["stream"]
            if (default_file and default_stream is not None):
                conf_list += ["[{type}:{file}:{stream}]".format(
                    type=m, file=default_file, stream=default_stream)]
        globals.log.debug("{fn}(): default config = "
                          "{c}".format(fn=fn, c=conf_list))
        config = parse_configuration_string("\n".join(conf_list))
    
    return config

def get_file_duration(file):
    """Calculate the duration a media file as a timedelta object."""
    fn = "get_file_duration"
    command = FFprobeCommand([file])
    globals.log.debug("{fn}(): {cmd}".format(fn=fn, cmd=command))
    # Only consider the first stream. If it's the only stream in the
    # file, great; otherwise it seems reasonable to assume that all
    # streams in the same file will have the same duration.
    ss, ms = command.get_entries(
        section="format", find_list=["duration"])[0].split(".")
    ms = ms[:3].ljust(3, "0")
    globals.log.debug("{fn}(): ss = {ss}, ms = {ms}".format(fn=fn, ss=ss,
                                                            ms=ms))
    return datetime.timedelta(seconds=int(ss), milliseconds=int(ms))

def make_new_segment(type, filename, punch_in, punch_out, num):
    """Make a new segment instance of the correct class."""
    fn = "make_new_segment"
    globals.log.debug("{fn}(): type = {t}".format(fn=fn, t=type))
    globals.log.debug("{fn}(): filename = {f}".format(fn=fn, f=filename))
    globals.log.debug("{fn}(): punch in = {i}".format(fn=fn, i=punch_in))
    globals.log.debug("{fn}(): punch out = {o}".format(fn=fn, o=punch_out))
    globals.log.debug("{fn}(): num = {n}".format(fn=fn, n=num))
    
    if (type == "audio"):
        return AudioSegment(file=filename, punch_in=punch_in,
                            punch_out=punch_out, input_stream=num)
    elif (type == "video"):
        return VideoSegment(file=filename, punch_in=punch_in,
                            punch_out=punch_out, input_stream=num)
    elif (type == "frame"):
        return FrameSegment(file=filename, punch_in=punch_in,
                            punch_out=punch_out, frame_number=num)
    else:
        return None

def process_timestamp_pair(args, times):
    """Constructs timedelta instances from a pair of config timestamps."""
    fn = "process_timestamp_pair"
    globals.log.debug("{fn}(): times[0] = {t}".format(fn=fn, t=times[0]))
    globals.log.debug("{fn}(): times[1] = {t}".format(fn=fn, t=times[1]))
    
    # If the first item in the timestamp list in the configuration file
    # is a filename, the parser inserts a zero timestamp before it. We
    # can therefore guarantee that the first item of the pair will
    # always be a timestamp.
    t0 = datetime.timedelta(
        hours=times[0]["hh"], minutes=times[0]["mm"],
        seconds=times[0]["ss"], milliseconds=times[0]["ms"])
    if (times[1]):
        if (len(times[1]) == 1): # filename
            t1 = t0 + get_file_duration(
                prefix_path(args.prefix, times[1]["filename"]))
        elif (len(times[1]) == 4): # normal timestamp
            t1 = datetime.timedelta(
                hours=times[1]["hh"], minutes=times[1]["mm"],
                seconds=times[1]["ss"], milliseconds=times[1]["ms"])
        else:
            globals.log.error("{fn}():unreadable timestamp {t}".format(
                fn=fn, t=times[1]))
            t1 = None
    else:
        t1 = None
    
    globals.log.debug("{fn}(): t0 = {t}".format(fn=fn, t=t0))
    globals.log.debug("{fn}(): t1 = {t}".format(fn=fn, t=t1))
    return t0, t1

def process_time_list(args, type, filename, num, time_list):
    """Process an audio or video stream and build a list of segments."""
    fn = "process_time_list"
    if (Path(filename).exists() and type in ["audio", "video"]):
        stream_duration = get_file_duration(filename)
    else:
        stream_duration = 0
    segments = []
    globals.log.debug("{fn}(): stream duration = {d}".format(
        fn=fn, d=stream_duration))
    
    # No timestamps: punch in at 0, out at stream duration.
    if (len(time_list) == 0):
        punch_in = datetime.timedelta()
        punch_out = stream_duration
        segments.append(make_new_segment(type, filename, punch_in,
                                         punch_out, num))
    else:
        # Process each pair of timestamps as punch in, out. If there's
        # an odd number of items, the last one is processed separately.
        for t in zip(time_list[::2], time_list[1::2]):
            punch_in, punch_out = process_timestamp_pair(args, t)
            if (punch_in == punch_out):
                globals.log.warning(
                    "punch in ({i}s) and punch out ({o}s) times are "
                    "equal; no segment will be "
                    "generated".format(i=punch_in.total_seconds(),
                                       o=punch_out.total_seconds()))
                continue
            elif (punch_out < punch_in):
                globals.log.error(
                    "punch out time ({i}s) falls before punch in time "
                    "({o}s); can't generate a valid "
                    "segment".format(i=punch_in.total_seconds(),
                                     o=punch_out.total_seconds()))
                sys.exit(1)
            segments.append(make_new_segment(type, filename, punch_in,
                                             punch_out, num))
    
    # Odd number of timestamps: punch in at last timestamp,
    # out at stream duration.
    if (len(time_list) % 2 != 0):
        globals.log.debug("{fn}(): odd number of timestamps".format(fn=fn))
        punch_in, _ = process_timestamp_pair(args, [time_list[-1], None])
        punch_out = stream_duration
        segments.append(make_new_segment(type, filename, punch_in,
                                         punch_out, num))
    return segments

def process_input_streams(args, config):
    """Process a list of stream specifications and build a list of segments."""
    fn = "process_input_streams"
    globals.log.info("Processing input streams...")
    segments = []
    for cnf in config:
        globals.log.debug("{fn}(): type = {t}".format(fn=fn, t=cnf["type"]))
        globals.log.debug(
            "{fn}(): filename = {f}".format(fn=fn, f=cnf["filename"]))
        globals.log.debug("{fn}(): num = {n}".format(fn=fn, n=cnf["num"]))
        globals.log.debug("{fn}(): times = {t}".format(fn=fn, t=cnf["times"]))
    
        segments += process_time_list(args, cnf["type"], cnf["filename"],
                                      cnf["num"], cnf["times"])  
    return segments

def smallest_video_dimensions(args, segments):
    """Compute the smallest frame dimensions across all video inputs."""
    fn = "smallest_video_dimensions"
    width = 2048
    height = 1536
    for s in segments:
        if isinstance(s, FrameSegment):
            continue
        command = FFprobeCommand([s.input_file])
        globals.log.debug("{fn}(): {cmd}".format(fn=fn, cmd=command))
        w, h = command.get_entries(find_list=["width", "height"])
        if (w * h) < (width * height):
            width, height = w, h
    return width, height

def process_frame_segments(args, segments, width, height):
    """Post-process frame segments to set frame images, etc."""
    fn = "process_frame_segments"
    globals.log.info("Processing frames...")
    frame_segments = [s for s in segments if isinstance(s, FrameSegment)]
    n = len(frame_segments)
    globals.log.debug("{fn}(): num frames = {n}".format(fn=fn, n=n))
    progress = ProgressBar(max_value=n,
                           quiet=args.quiet or args.debug or n == 0)
    progress.update(0)
    for i, f in enumerate(frame_segments):
        try:
            globals.log.debug(
                "{fn}(): frame (before) = {b}".format(fn=fn, b=f))
            # Frame segments that use a frame from the previous segment.
            if (f.input_file == "^"):
                if (f.segment_number > 0):
                    prev = segments[f.segment_number - 1]
                    globals.log.debug(
                        "{fn}(): prev = {p}".format(fn=fn, p=prev))
                    prev.generate_temp_file(args.output, width=width,
                                            height=height)
                    f.use_frame(
                        prev.generate_frame(f.frame_number, args.output,
                                            width=width, height=height))
                else:
                    globals.log.error(
                        "frame segment {s} is attempting to use the last "
                        "frame of a non-existent previous "
                        "segment".format(s=f.segment_number))
                    sys.exit(1)
            # Frame segments whose frame comes from a PDF file.
            else:
                suffix = PurePath(f.input_file).suffix
                if (suffix.lower() == ".pdf"):
                    f.use_frame(f.generate_temp_file(args.output, width=width,
                                            height=height))
                else:
                    globals.log.error(
                        'unexpected input file type "{s}" for frame segment '
                        "{f}".format(s=suffix, f=f.segment_number))
                    sys.exit(1)
            progress.update(i)
            globals.log.debug("{fn}(): frame (after) = ""{a}".format(fn=fn, a=f))
        except SegmentError as e:
            progress.finish()
            globals.log.exception(e)
            sys.exit(1)
    else:
        progress.finish()

def render_podcast(args, audio_segments, video_segments, output, duration):
    """Stitch together the various input components into the final podcast."""
    fn = "render_podcast"
    globals.log.info("Rendering final podcast...")
    command = FFmpegConcatCommand(has_audio=len(audio_segments) > 0,
                                  has_video=len(video_segments) > 0,
                                  max_progress=duration,
                                  quiet=args.quiet and not args.debug,
                                  process_audio=args.process_audio,
                                  process_video=args.process_video,
                                  audio_codec=args.audio_codec,
                                  video_codec=args.video_codec)
    input_files = Segment.input_files()
    for f in input_files:
        if (input_files[f]):
            command.append_input_options(input_files[f])
        command.append_input_options(["-i", f])
    for s in (audio_segments + video_segments):
        command.append_filter(s.trim_filter())
    command.append_concat_filter("a", [s for s in audio_segments])
    if (args.normalise):
        command.append_normalisation_filter()
    command.append_concat_filter("v", [s for s in video_segments])
    if args.preview:
        globals.log.info("PREVIEW MODE: {fps} fps".format(fps=args.preview))
        command.append_output_options(["-r", args.preview])
    command.append_output_options([output])
    globals.log.debug("{fn}(): {c}".format(fn=fn, c=command))
    if (command.run() != 0):
        globals.log.error("Failed to render final podcast")

def cleanup(segments):
    """Clean up generated temporary files."""
    globals.log.info("Cleaning up...")
    for s in segments:
        s.delete_temp_files()

def main():
    fn = "main"
    logging.basicConfig(
        level=logging.INFO,
        format="%(levelname)s: {p}: %(message)s".format(p=globals.PROGRAM))
    segments = None
    
    try:
        args = parse_command_line()
        check_arguments(args)
    
        config = get_configuration(args)
    
        segments = process_input_streams(args, config)
        globals.log.debug("{fn}(): audio segments = {a}".format(
            fn=fn, a=[s for s in segments if isinstance(s, AudioSegment)]))
        globals.log.debug("{fn}(): video segments = {v}".format(
            fn=fn, v=[s for s in segments if isinstance(s, VideoSegment)]))
    
        audio_segments = [s for s in segments if isinstance(s, AudioSegment)]
        video_segments = [s for s in segments if isinstance(s, VideoSegment)]
    
        audio_duration = sum([s.get_duration() for s in audio_segments])
        video_duration = sum([s.get_duration() for s in video_segments])
        globals.log.debug("{fn}(): audio duration = "
                          "{a}".format(fn=fn, a=audio_duration))
        globals.log.debug("{fn}(): video duration = "
                          "{v}".format(fn=fn, v=video_duration))
    
        if (len(audio_segments) and len(video_segments)):
            if (audio_duration != video_duration):
                globals.log.warning("total video duration ({v}s) doesn't match "
                            "total audio duration "
                            "({a}s)".format(v=video_duration, a=audio_duration))
        
        width, height = smallest_video_dimensions(args, video_segments)
        globals.log.debug("{fn}(): width = {w}, height = "
                          "{h}".format(fn=fn, w=width, h=height))
        
        process_frame_segments(args, segments, width, height)
    
        globals.log.debug("{fn}(): input files = "
                          "{i}".format(fn=fn, i=Segment.input_files()))
    
        render_podcast(args, audio_segments, video_segments, args.output,
                       max(audio_duration, video_duration))

except (KeyboardInterrupt):
        pass
    finally:
        if segments and not args.keep:
            cleanup(segments)

if (__name__ == "__main__"):
    assert sys.version_info >= (3, 5)
    main()

#!/usr/bin/env python3

import argparse
import datetime
import logging
from pathlib import Path, PurePath
import sys

from pyparsing import ParseResults

args = parser.parse_args()
    
    return args

def cleanup(segments):
    """Clean up generated temporary files."""
    globals.log.info("Cleaning up...")
    for s in segments:
        s.delete_temp_files()

except (KeyboardInterrupt):
        pass
    finally:
        if segments and not args.keep:
            cleanup(segments)

if (__name__ == "__main__"):
    assert sys.version_info >= (3, 7)
    main()

Ignore Space Show notes View shell_command/shell_command.py
import datetime import json import tempfile from pathlib import Path import re import shutil import pexpect from progress_bar import ProgressBar class ShellCommand(object): """A shell command. _executable contains the full path to the relevant executable. _base_options is a list of standard general options for this command. """ _executable = "" _base_options = [] _expect_patterns = [] @staticmethod def shellquote(s): """Quote a string so it can be safely pasted into the shell.""" # Note: pipes/shlex.quote() only wraps '' around things, # it doesn't do things like \( \), which we also need. # We double-quote everything because that makes it easier # to deal with single quotes. if s: regexes = [ # grouping parentheses for convert re.compile(r"^(\(\|\))$"), # double quote (in middle of string) re.compile(r'^(.+".)$'), # quotes around entire string re.compile(r"""^((?P<quote>['"]).(?P=quote))$"""), # command line switch (starts with one or two "-") re.compile(r"^(--?.+)$"), # command path (starts with "/") re.compile(r"^(/[\w/]+)$")] substitutions = [r"\\\1", r"'\1'", r"\1", r"\1", r"\1"] num_matches = 0 for subst in zip(regexes, substitutions): (s, n) = subst[0].subn(subst[1], s) num_matches += n if num_matches == 0: # catch-all for non-word characters when none of the other # substitutions apply s = re.compile(r"^(.\W+.)$").sub(r'"\1"', s) return s def __init__(self, input_options=[], output_options=[], quiet=False): self.input_options = input_options self.output_options = output_options self.progress = None self.process = None def __repr__(self): return "<{cls}: {cmd}>".format( cls=self.__class__.__name__, cmd=self.command_string(quote=True)) def append_input_options(self, items=[]): """Add a list of items to the end of the input options.""" self.input_options += items def prepend_input_options(self, items=[]): """Add a list of items at the front of the input options.""" self.input_options = items + self.input_options def append_output_options(self, items=[]): """Add a list of items to the end of the output options.""" self.output_options += items def prepend_output_options(self, items=[]): """Add a list of items at the front of the output options.""" self.output_options = items + self.output_options def executable_string(self, quote=False): """Return the executable as a string.""" if quote: return ShellCommand.shellquote(self._executable) else: return self._executable def argument_string(self, quote=False): """Return the list of arguments as a string.""" args = self._base_options + self.input_options + self.output_options if quote: return " ".join([ShellCommand.shellquote(a) for a in args]) else: return " ".join(args) def argument_list(self): """Return a combined list of all arguments.""" return self._base_options + self.input_options + self.output_options def command_string(self, quote=False): """Return the entire command as a string.""" return "{exe} {arg}".format(exe=self.executable_string(quote), arg=self.argument_string(quote)) def process_pattern(self, pat): """Respond to a pexpect pattern. Return True on EOF.""" return (pat == 0) def run(self): """Execute the command in a subprocess.""" self.process = pexpect.spawn(self.executable_string(), self.argument_list()) # EOF is always the first pattern. patterns = self.process.compile_pattern_list( [pexpect.EOF] + self._expect_patterns) try: while True: i = self.process.expect_list(patterns, timeout=None) if self.process_pattern(i): break except (KeyboardInterrupt): pass finally: if self.progress: self.progress.finish() self.process.close() return self.process.exitstatus def get_output(self): """Execute the command in a subprocess and return the output.""" return pexpect.run(self.command_string(quote=True)) class ConvertCommand(ShellCommand): """An ImageMagick convert command.""" _executable = shutil.which("convert") _base_options = ["-density", "600", "xc:dimgrey", "null:", # dark grey background "("] def __init__(self, input_options=[], output_options=[], width=2048, height=1536): super().__init__(input_options, output_options) self._base_options = (["-size", "{w}x{h}".format(w=width, h=height)] + self._base_options) self.append_input_options( ["-resize", "{w}x{h}".format(w=width, h=height), "-background", "white", "-alpha", "remove", "-type", "truecolor", # force RGB (this and next line) "-define", "colorspace:auto-grayscale=off"]) self.prepend_output_options([")", "-gravity", "center", "-layers", "composite", "-flatten"]) class FFprobeCommand(ShellCommand): """An ffprobe shell command.""" _executable = shutil.which("ffprobe") _base_options = ["-loglevel", "error", "-show_entries", "format:stream", "-print_format", "json"] def __init__(self, input_options=[], output_options=[]): super().__init__(input_options, output_options) self.entries = None # The input file should be the last input option. assert(Path(self.input_options[-1]).exists()) self.last_modified = Path(self.input_options[-1]).stat().st_mtime def get_entries(self, section="stream", find_list=[]): """Fetch specified attributes from the input file.""" # Re-fetch if the file's changed since we last looked. modified = Path(self.input_options[-1]).stat().st_mtime if (not self.entries) or (modified > self.last_modified): js = json.loads(self.get_output()) self.entries = {"format": js["format"], "stream": js["streams"][0]} return [self.entries[section][f] for f in find_list] class FFmpegCommand(ShellCommand): """A "simple" ffmpeg shell command.""" _executable = shutil.which("ffmpeg") _base_options = ["-y", "-nostdin"] class FFmpegConcatCommand(FFmpegCommand): """An ffmpeg shell command with a complex concat filter.""" _expect_patterns = [r"time=(\d\d):(\d\d):(\d\d\.\d\d)"] def __init__(self, input_options=[], output_options=[], quiet=False, max_progress=100, has_audio=False, has_video=False, process_audio=True, process_video=True, audio_codec="pcm_s16le", video_codec="h264"): super().__init__(input_options, output_options) self.progress = ProgressBar(max_value=max_progress, quiet=quiet) self.has_video = has_video self.process_video = process_video self.video_codec = video_codec if (self.has_video): self.prepend_output_options(["-map", "[vconc]"]) if (self.process_video): self.prepend_output_options(["-pix_fmt", "yuv420p"]) self.prepend_output_options(["-codec:v", self.video_codec]) self.has_audio = has_audio self.process_audio = process_audio self.audio_codec = audio_codec if (self.has_audio): if (self.process_audio): self.prepend_output_options(["-ac", "1", "-map", "[anorm]"]) else: self.prepend_output_options(["-map", "[aconc]"]) self.prepend_output_options(["-codec:a", self.audio_codec]) self.filters = [] def append_filter(self, filter): """Append a filter to the filters list.""" if (filter): self.filters.append(filter) def append_normalisation_filter(self): """Append a normalisation audio filter to the complex filter.""" if (self.has_audio): self.append_filter("[aconc] dynaudnorm=r=0.25:f=10:b=1 [anorm]") def append_concat_filter(self, frame_type, segments=[]): """Append a concat filter to the filters list""" # Ignore frame segments. if frame_type in ["a", "v"]: if (len(segments) > 1): self.append_filter( "{inspecs} concat=n={n}:v={v}:a={a} [{t}conc]".format( inspecs=" ".join([s.output_stream_specifier() for s in segments]), n=len(segments), v=int(frame_type == "v"), a=int(frame_type == "a"), t=frame_type)) elif (len(segments) == 1): self.append_filter( "{inspec} {a}null [{t}conc]".format( inspec=segments[0].output_stream_specifier(), a=frame_type if frame_type == "a" else "", t=frame_type)) def build_complex_filter(self): """Build the complete complex filter. Filters in the filtergraph are separated by ";". """ return "{f}".format(f=";".join(self.filters)) def argument_string(self, quote=False): """Return the list of arguments as a string.""" args = (self._base_options + self.input_options + ["-filter_complex", self.build_complex_filter()] + self.output_options) if quote: return " ".join([ShellCommand.shellquote(a) for a in args]) else: return " ".join(args) def argument_list(self): """Return a combined list of all arguments.""" return (self._base_options + self.input_options + ["-filter_complex", self.build_complex_filter()] + self.output_options) def process_pattern(self, pat): """Respond to a pexpect pattern. Return True on EOF.""" if (pat == 1): elapsed = datetime.timedelta( hours=int(self.process.match.group(1)), minutes=int(self.process.match.group(2)), seconds=float(self.process.match.group(3))) self.progress.update(elapsed.total_seconds()) return (pat == 0) import datetime import json import tempfile from pathlib import Path import re import shutil import pexpect from progress_bar import ProgressBar class ShellCommand(object): """A shell command. _executable contains the full path to the relevant executable. _base_options is a list of standard general options for this command. """ _executable = "" _base_options = [] _expect_patterns = [] @staticmethod def shellquote(s): """Quote a string so it can be safely pasted into the shell.""" # Note: pipes/shlex.quote() only wraps '' around things, # it doesn't do things like \( \), which we also need. # We double-quote everything because that makes it easier # to deal with single quotes. if s: regexes = [ # grouping parentheses for convert re.compile(r"^(\(\|\))$"), # double quote (in middle of string) re.compile(r'^(.+".)$'), # quotes around entire string re.compile(r"""^((?P<quote>['"]).(?P=quote))$"""), # command line switch (starts with one or two "-") re.compile(r"^(--?.+)$"), # command path (starts with "/") re.compile(r"^(/[\w/]+)$")] substitutions = [r"\\\1", r"'\1'", r"\1", r"\1", r"\1"] num_matches = 0 for subst in zip(regexes, substitutions): (s, n) = subst[0].subn(subst[1], s) num_matches += n if num_matches == 0: # catch-all for non-word characters when none of the other # substitutions apply s = re.compile(r"^(.\W+.)$").sub(r'"\1"', s) return s def __init__(self, input_options=[], output_options=[], quiet=False): self.input_options = input_options self.output_options = output_options self.progress = None self.process = None def __repr__(self): return "<{cls}: {cmd}>".format( cls=self.__class__.__name__, cmd=self.command_string(quote=True)) def append_input_options(self, items=[]): """Add a list of items to the end of the input options.""" self.input_options += items def prepend_input_options(self, items=[]): """Add a list of items at the front of the input options.""" self.input_options = items + self.input_options def append_output_options(self, items=[]): """Add a list of items to the end of the output options.""" self.output_options += items def prepend_output_options(self, items=[]): """Add a list of items at the front of the output options.""" self.output_options = items + self.output_options def executable_string(self, quote=False): """Return the executable as a string.""" if quote: return ShellCommand.shellquote(self._executable) else: return self._executable def argument_string(self, quote=False): """Return the list of arguments as a string.""" args = self._base_options + self.input_options + self.output_options if quote: return " ".join([ShellCommand.shellquote(a) for a in args]) else: return " ".join(args) def argument_list(self): """Return a combined list of all arguments.""" return self._base_options + self.input_options + self.output_options def command_string(self, quote=False): """Return the entire command as a string.""" return "{exe} {arg}".format(exe=self.executable_string(quote), arg=self.argument_string(quote)) def process_pattern(self, pat): """Respond to a pexpect pattern. Return True on EOF.""" return (pat == 0) def run(self): """Execute the command in a subprocess.""" self.process = pexpect.spawn(self.executable_string(), self.argument_list()) # EOF is always the first pattern. patterns = self.process.compile_pattern_list( [pexpect.EOF] + self._expect_patterns) try: while True: i = self.process.expect_list(patterns, timeout=None) if self.process_pattern(i): break except (KeyboardInterrupt): pass finally: if self.progress: self.progress.finish() self.process.close() return self.process.exitstatus def get_output(self): """Execute the command in a subprocess and return the output.""" return pexpect.run(self.command_string(quote=True)) class ConvertCommand(ShellCommand): """An ImageMagick convert command.""" _executable = shutil.which("convert") _base_options = ["-density", "600", "xc:dimgrey", "null:", # dark grey background "("] def __init__(self, input_options=[], output_options=[], width=2048, height=1536): super().__init__(input_options, output_options) self._base_options = (["-size", "{w}x{h}".format(w=width, h=height)] + self._base_options) self.append_input_options( ["-resize", "{w}x{h}".format(w=width, h=height), "-background", "white", "-alpha", "remove", "-type", "truecolor", # force RGB (this and next line) "-define", "colorspace:auto-grayscale=off"]) self.prepend_output_options([")", "-gravity", "center", "-layers", "composite", "-flatten"]) class FFprobeCommand(ShellCommand): """An ffprobe shell command.""" _executable = shutil.which("ffprobe") _base_options = ["-loglevel", "error", "-show_entries", "format:stream", "-print_format", "json"] def __init__(self, input_options=[], output_options=[]): super().__init__(input_options, output_options) self.entries = None # The input file should be the last input option. assert(Path(self.input_options[-1]).exists()) self.last_modified = Path(self.input_options[-1]).stat().st_mtime def get_entries(self, section="stream", find_list=[]): """Fetch specified attributes from the input file.""" # Re-fetch if the file's changed since we last looked. modified = Path(self.input_options[-1]).stat().st_mtime if (not self.entries) or (modified > self.last_modified): js = json.loads(self.get_output()) self.entries = {"format": js["format"], "stream": js["streams"][0]} return [self.entries[section][f] for f in find_list] class FFmpegCommand(ShellCommand): """A "simple" ffmpeg shell command.""" _executable = shutil.which("ffmpeg") _base_options = ["-y", "-nostdin"] class FFmpegConcatCommand(FFmpegCommand): """An ffmpeg shell command with a complex concat filter.""" _expect_patterns = [r"time=(\d\d):(\d\d):(\d\d\.\d\d)"] def __init__(self, input_options=[], output_options=[], quiet=False, max_progress=100, has_audio=False, has_video=False, process_audio=True, process_video=True, audio_codec="pcm_s16le", video_codec="h264"): super().__init__(input_options, output_options) self.progress = ProgressBar(max_value=max_progress, quiet=quiet) self.has_video = has_video self.process_video = process_video self.video_codec = video_codec if (self.has_video): self.prepend_output_options(["-map", "[vconc]"]) if (self.process_video): self.prepend_output_options(["-pix_fmt", "yuv420p"]) self.prepend_output_options(["-codec:v", self.video_codec]) self.has_audio = has_audio self.process_audio = process_audio self.audio_codec = audio_codec if (self.has_audio): if (self.process_audio): self.prepend_output_options(["-ac", "1", "-map", "[anorm]"]) else: self.prepend_output_options(["-map", "[aconc]"]) self.prepend_output_options(["-codec:a", self.audio_codec]) self.filters = [] def append_filter(self, filter): """Append a filter to the filters list.""" if (filter): self.filters.append(filter) def append_normalisation_filter(self): """Append a normalisation audio filter to the complex filter.""" if (self.has_audio): self.append_filter("[aconc] dynaudnorm=r=0.25:f=10:b=y [anorm]") def append_concat_filter(self, frame_type, segments=[]): """Append a concat filter to the filters list""" # Ignore frame segments. if frame_type in ["a", "v"]: if (len(segments) > 1): self.append_filter( "{inspecs} concat=n={n}:v={v}:a={a} [{t}conc]".format( inspecs=" ".join([s.output_stream_specifier() for s in segments]), n=len(segments), v=int(frame_type == "v"), a=int(frame_type == "a"), t=frame_type)) elif (len(segments) == 1): self.append_filter( "{inspec} {a}null [{t}conc]".format( inspec=segments[0].output_stream_specifier(), a=frame_type if frame_type == "a" else "", t=frame_type)) def build_complex_filter(self): """Build the complete complex filter. Filters in the filtergraph are separated by ";". """ return "{f}".format(f=";".join(self.filters)) def argument_string(self, quote=False): """Return the list of arguments as a string.""" args = (self._base_options + self.input_options + ["-filter_complex", self.build_complex_filter()] + self.output_options) if quote: return " ".join([ShellCommand.shellquote(a) for a in args]) else: return " ".join(args) def argument_list(self): """Return a combined list of all arguments.""" return (self._base_options + self.input_options + ["-filter_complex", self.build_complex_filter()] + self.output_options) def process_pattern(self, pat): """Respond to a pexpect pattern. Return True on EOF.""" if (pat == 1): elapsed = datetime.timedelta( hours=int(self.process.match.group(1)), minutes=int(self.process.match.group(2)), seconds=float(self.process.match.group(3))) self.progress.update(elapsed.total_seconds()) return (pat == 0)

Ignore Space Show notes View

shell_command/shell_command.py

import datetime
import json
import tempfile
from pathlib import Path
import re
import shutil

import pexpect

from progress_bar import ProgressBar

class ShellCommand(object):
    """A shell command.
    
    _executable contains the full path to the relevant executable.
    
    _base_options is a list of standard general options for this
    command.
    """
    _executable = ""
    _base_options = []
    _expect_patterns = []
    
    @staticmethod
    def shellquote(s):
        """Quote a string so it can be safely pasted into the shell."""
        # Note: pipes/shlex.quote() only wraps '' around things,
        # it doesn't do things like , which we also need.
        # We double-quote everything because that makes it easier
        # to deal with single quotes.
        if s:
            regexes = [
                # grouping parentheses for convert
                re.compile(r"^($|$)$"),
                # double quote (in middle of string)
                re.compile(r'^(.+".*)$'),
                # quotes around entire string
                re.compile(r"""^((?P<quote>['"]).*(?P=quote))$"""),
                # command line switch (starts with one or two "-")
                re.compile(r"^(--?.+)$"),
                # command path (starts with "/")
                re.compile(r"^(/[\w/]+)$")]
            substitutions = [r"\\\1", r"'\1'", r"\1", r"\1", r"\1"]
            num_matches = 0
            for subst in zip(regexes, substitutions):
                (s, n) = subst[0].subn(subst[1], s)
                num_matches += n
            if num_matches == 0:
                # catch-all for non-word characters when none of the other
                # substitutions apply
                s = re.compile(r"^(.*\W+.*)$").sub(r'"\1"', s)
        return s
    
    def __init__(self, input_options=[], output_options=[], quiet=False):
        self.input_options = input_options
        self.output_options = output_options
        self.progress = None
        self.process = None
    
    def __repr__(self):
        return "<{cls}: {cmd}>".format(
            cls=self.__class__.__name__,
            cmd=self.command_string(quote=True))

def append_input_options(self, items=[]):
        """Add a list of items to the end of the input options."""
        self.input_options += items
    
    def prepend_input_options(self, items=[]):
        """Add a list of items at the front of the input options."""
        self.input_options = items + self.input_options
    
    def append_output_options(self, items=[]):
        """Add a list of items to the end of the output options."""
        self.output_options += items
    
    def prepend_output_options(self, items=[]):
        """Add a list of items at the front of the output options."""
        self.output_options = items + self.output_options
    
    def executable_string(self, quote=False):
        """Return the executable as a string."""
        if quote:
            return ShellCommand.shellquote(self._executable)
        else:
            return self._executable
    
    def argument_string(self, quote=False):
        """Return the list of arguments as a string."""
        args = self._base_options + self.input_options + self.output_options
        if quote:
            return " ".join([ShellCommand.shellquote(a) for a in args])
        else:
            return " ".join(args)
    
    def argument_list(self):
        """Return a combined list of all arguments."""
        return self._base_options + self.input_options + self.output_options
    
    def command_string(self, quote=False):
        """Return the entire command as a string."""
        return "{exe} {arg}".format(exe=self.executable_string(quote),
                                    arg=self.argument_string(quote))
    
    def process_pattern(self, pat):
        """Respond to a pexpect pattern. Return True on EOF."""
        return (pat == 0)
    
    def run(self):
        """Execute the command in a subprocess."""
        self.process = pexpect.spawn(self.executable_string(),
                                     self.argument_list())
        # EOF is *always* the first pattern.
        patterns = self.process.compile_pattern_list(
            [pexpect.EOF] + self._expect_patterns)
        try:
            while True:
                i = self.process.expect_list(patterns, timeout=None)
                if self.process_pattern(i):
                    break
        except (KeyboardInterrupt):
            pass
        finally:
            if self.progress:
                self.progress.finish()
            self.process.close()
            return self.process.exitstatus
    
    def get_output(self):
        """Execute the command in a subprocess and return the output."""
        return pexpect.run(self.command_string(quote=True))

class ConvertCommand(ShellCommand):
    """An ImageMagick convert command."""
    _executable = shutil.which("convert")
    _base_options = ["-density", "600",
                     "xc:dimgrey", "null:", # dark grey background
                     "("]
    
    def __init__(self, input_options=[], output_options=[], width=2048, height=1536):
        super().__init__(input_options, output_options)
        self._base_options = (["-size", "{w}x{h}".format(w=width, h=height)] +
                              self._base_options)
        self.append_input_options(
            ["-resize", "{w}x{h}".format(w=width, h=height),
             "-background", "white", 
             "-alpha", "remove",
             "-type", "truecolor", # force RGB (this and next line)
             "-define", "colorspace:auto-grayscale=off"])
        self.prepend_output_options([")",
                                     "-gravity", "center",
                                     "-layers", "composite",
                                     "-flatten"])

class FFprobeCommand(ShellCommand):
    """An ffprobe shell command."""
    _executable = shutil.which("ffprobe")
    _base_options = ["-loglevel", "error",
                     "-show_entries", "format:stream",
                     "-print_format", "json"]
    
    def __init__(self, input_options=[], output_options=[]):
        super().__init__(input_options, output_options)
        self.entries = None
        # The input file should be the last input option.
        assert(Path(self.input_options[-1]).exists())
        self.last_modified = Path(self.input_options[-1]).stat().st_mtime
    
    def get_entries(self, section="stream", find_list=[]):
        """Fetch specified attributes from the input file."""
        # Re-fetch if the file's changed since we last looked.
        modified = Path(self.input_options[-1]).stat().st_mtime
        if (not self.entries) or (modified > self.last_modified):
            js = json.loads(self.get_output())
            self.entries = {"format": js["format"], "stream": js["streams"][0]}
        return [self.entries[section][f] for f in find_list]

class FFmpegCommand(ShellCommand):
    """A "simple" ffmpeg shell command."""
    _executable = shutil.which("ffmpeg")
    _base_options = ["-y", "-nostdin"]

class FFmpegConcatCommand(FFmpegCommand):
    """An ffmpeg shell command with a complex concat filter."""
    _expect_patterns = [r"time=(\d\d):(\d\d):(\d\d\.\d\d)"]
    
    def __init__(self, input_options=[], output_options=[], quiet=False,
                 max_progress=100, has_audio=False, has_video=False,
                 process_audio=True, process_video=True,
                 audio_codec="pcm_s16le", video_codec="h264"):
        super().__init__(input_options, output_options)
        self.progress = ProgressBar(max_value=max_progress, quiet=quiet)
        self.has_video = has_video
        self.process_video = process_video
        self.video_codec = video_codec
        if (self.has_video):
            self.prepend_output_options(["-map", "[vconc]"])
            if (self.process_video):
                self.prepend_output_options(["-pix_fmt", "yuv420p"])
            self.prepend_output_options(["-codec:v", self.video_codec])
        self.has_audio = has_audio
        self.process_audio = process_audio
        self.audio_codec = audio_codec
        if (self.has_audio):
            if (self.process_audio):
                self.prepend_output_options(["-ac", "1",
                                             "-map", "[anorm]"])
            else:
                self.prepend_output_options(["-map", "[aconc]"])
            self.prepend_output_options(["-codec:a", self.audio_codec])
        self.filters = []
    
    def append_filter(self, filter):
        """Append a filter to the filters list."""
        if (filter):
            self.filters.append(filter)
        
    def append_normalisation_filter(self):
        """Append a normalisation audio filter to the complex filter."""
        if (self.has_audio):
            self.append_filter("[aconc] dynaudnorm=r=0.25:f=10:b=1 [anorm]")
    
    def append_concat_filter(self, frame_type, segments=[]):
        """Append a concat filter to the filters list"""
        # Ignore frame segments.
        if frame_type in ["a", "v"]:
            if (len(segments) > 1):
                self.append_filter(
                    "{inspecs} concat=n={n}:v={v}:a={a} [{t}conc]".format(
                        inspecs=" ".join([s.output_stream_specifier()
                                        for s in segments]),
                        n=len(segments), v=int(frame_type == "v"),
                        a=int(frame_type == "a"), t=frame_type))
            elif (len(segments) == 1):
                self.append_filter(
                    "{inspec} {a}null [{t}conc]".format(
                        inspec=segments[0].output_stream_specifier(),
                        a=frame_type if frame_type == "a" else "",
                        t=frame_type))
        
    def build_complex_filter(self):
        """Build the complete complex filter.
        
        Filters in the filtergraph are separated by ";".
        """
        return "{f}".format(f=";".join(self.filters))
    
    def argument_string(self, quote=False):
        """Return the list of arguments as a string."""
        args = (self._base_options + self.input_options +
                ["-filter_complex", self.build_complex_filter()] +
                self.output_options)
        if quote:
            return " ".join([ShellCommand.shellquote(a) for a in args])
        else:
            return " ".join(args)
    
    def argument_list(self):
        """Return a combined list of all arguments."""
        return (self._base_options + self.input_options +
                ["-filter_complex", self.build_complex_filter()] +
                self.output_options)
    
    def process_pattern(self, pat):
        """Respond to a pexpect pattern. Return True on EOF."""
        if (pat == 1):
            elapsed = datetime.timedelta(
                hours=int(self.process.match.group(1)),
                minutes=int(self.process.match.group(2)),
                seconds=float(self.process.match.group(3)))
            self.progress.update(elapsed.total_seconds())
        return (pat == 0)

import datetime
import json
import tempfile
from pathlib import Path
import re
import shutil

import pexpect

from progress_bar import ProgressBar

class FFmpegCommand(ShellCommand):
    """A "simple" ffmpeg shell command."""
    _executable = shutil.which("ffmpeg")
    _base_options = ["-y", "-nostdin"]

class FFmpegConcatCommand(FFmpegCommand):
    """An ffmpeg shell command with a complex concat filter."""
    _expect_patterns = [r"time=(\d\d):(\d\d):(\d\d\.\d\d)"]
    
    def __init__(self, input_options=[], output_options=[], quiet=False,
                 max_progress=100, has_audio=False, has_video=False,
                 process_audio=True, process_video=True,
                 audio_codec="pcm_s16le", video_codec="h264"):
        super().__init__(input_options, output_options)
        self.progress = ProgressBar(max_value=max_progress, quiet=quiet)
        self.has_video = has_video
        self.process_video = process_video
        self.video_codec = video_codec
        if (self.has_video):
            self.prepend_output_options(["-map", "[vconc]"])
            if (self.process_video):
                self.prepend_output_options(["-pix_fmt", "yuv420p"])
            self.prepend_output_options(["-codec:v", self.video_codec])
        self.has_audio = has_audio
        self.process_audio = process_audio
        self.audio_codec = audio_codec
        if (self.has_audio):
            if (self.process_audio):
                self.prepend_output_options(["-ac", "1",
                                             "-map", "[anorm]"])
            else:
                self.prepend_output_options(["-map", "[aconc]"])
            self.prepend_output_options(["-codec:a", self.audio_codec])
        self.filters = []
    
    def append_filter(self, filter):
        """Append a filter to the filters list."""
        if (filter):
            self.filters.append(filter)
        
    def append_normalisation_filter(self):
        """Append a normalisation audio filter to the complex filter."""
        if (self.has_audio):
            self.append_filter("[aconc] dynaudnorm=r=0.25:f=10:b=y [anorm]")
    
    def append_concat_filter(self, frame_type, segments=[]):
        """Append a concat filter to the filters list"""
        # Ignore frame segments.
        if frame_type in ["a", "v"]:
            if (len(segments) > 1):
                self.append_filter(
                    "{inspecs} concat=n={n}:v={v}:a={a} [{t}conc]".format(
                        inspecs=" ".join([s.output_stream_specifier()
                                        for s in segments]),
                        n=len(segments), v=int(frame_type == "v"),
                        a=int(frame_type == "a"), t=frame_type))
            elif (len(segments) == 1):
                self.append_filter(
                    "{inspec} {a}null [{t}conc]".format(
                        inspec=segments[0].output_stream_specifier(),
                        a=frame_type if frame_type == "a" else "",
                        t=frame_type))
        
    def build_complex_filter(self):
        """Build the complete complex filter.
        
        Filters in the filtergraph are separated by ";".
        """
        return "{f}".format(f=";".join(self.filters))
    
    def argument_string(self, quote=False):
        """Return the list of arguments as a string."""
        args = (self._base_options + self.input_options +
                ["-filter_complex", self.build_complex_filter()] +
                self.output_options)
        if quote:
            return " ".join([ShellCommand.shellquote(a) for a in args])
        else:
            return " ".join(args)
    
    def argument_list(self):
        """Return a combined list of all arguments."""
        return (self._base_options + self.input_options +
                ["-filter_complex", self.build_complex_filter()] +
                self.output_options)
    
    def process_pattern(self, pat):
        """Respond to a pexpect pattern. Return True on EOF."""
        if (pat == 1):
            elapsed = datetime.timedelta(
                hours=int(self.process.match.group(1)),
                minutes=int(self.process.match.group(2)),
                seconds=float(self.process.match.group(3)))
            self.progress.update(elapsed.total_seconds())
        return (pat == 0)

Show line notes below