MIPoPS Microservices

One liners and automation tools from Moving Image Preservation of Puget Sound

This page is an evolving library of some of the most common commands used for manipulating digital objects at Moving Image Preservation of Puget Sound.

The following commands are primarily written in bash. We've also included the most common FFmpeg commands we use. FFmpeg is a powerful tool for manipulating audiovisual files. For an excellent, in-depth resource for using FFmpeg in the command line, check out ffmprovsir.

Contributors: Ari Lavigne, Libby Hopfauf, Andrew Weaver, and everyone who contributed to the ffmprovisor project (where the ffmpeg commands presented here were sourced).

Creative Commons License This work is licensed under a Creative Commons Attribution 4.0 International License.

Moving and Copying Files

This is a loop script that will find all .MP4s in a directory and copy them into a new directory elsewhere. Useful for copying all access files from a directory and putting them into an Access copies folder.

First, cd into the directory holding the access files you want to copy. At MIPoPS, this is usually the Participating Institutions directory.

cd /MIPoPS/Desktop/PIs/Institution1

#!/bin/bash
while read TARGET
do rsync -av "$TARGET" /EXAMPLE/PATH/ACCESSFILES
done < <(find "$PWD" -iname "*.mp4")

This script could be modified to look for and move other file types by modifing the input in "*.mp4". Replacing that with "*.dv" would return all files ending in .dv. "SAM_2019_*.*" would move all files that have the naming convention "SAM_2019" of any file type. The asterix is a wildcard.

This script runs a loop that identifies all MP4s in a directory, then moves them into directories that have the same name.

At MIPoPS, this is useful for moving all access copies into their home directories..

cd EXAMPLE/PATH/TO/TARGET/DIR

At MIPoPS, this would usually be: cd /MIPoPS/Desktop/PIs/Institution1

#!/bin/bash
for f in *.mp4; do
[[ -f "$f" ]] || continue
dir="${f%.*}"
mv "$f" "$dir"
done

Change container (rewrap)

Rewrap a file

ffmpeg -i /PATH/TO/input_file.ext -c copy -map 0 /PATH/TO/output_file.ext

This script will rewrap a video file. It will create a new video video file where the inner content (the video, audio, and subtitle data) of the original file is unchanged, but these streams are rehoused within a different container format.

ffmpeg
starts the command
-i /PATH/TO/input_file.ext
path and name of the input file. To copy the full path of a file on a Mac, right-click on the file, then, while in the right-click menu, hold down the "OPTION" key to reveal the "Copy as Pathname" option.
-c copy
copy the streams directly, without re-encoding.
-map 0
map all streams of the input to the output.
By default, FFmpeg will only map one stream of each type (video, audio, subtitles) to the output file. However, files may have multiple streams of a given type - for example, a video may have several audio tracks for different languages. Therefore, if you want to preserve all the streams in the original, it's necessary to use this option.
/PATH/TO/output_file.ext
path and name of the output file. To copy the full path of a file on a Mac, right-click on the file, then, while in the right-click menu, hold down the "OPTION" key to reveal the "Copy as Pathname" option.
The new container you are rewrapping to is defined by the filename extension used here, e.g. .mkv, .mp4, .mov.
This section taken from ffimprovisr

Rewrap DV video to .dv file

ffmpeg -i /PATH/TO/input_file -f rawvideo -c:v copy /PATH/TO/output_file.dv

This script will take a video that is encoded in the DV Codec but wrapped in a different container (such as MOV) and rewrap it into a raw DV file (with the .dv extension). Since DV files potentially contain a great deal of provenance metadata within the DV stream, it is necessary to rewrap files in this method to avoid unintentional stripping of this metadata.

ffmpeg
starts the command
-i /PATH/TO/input_file
path and name of the input file. To copy the full path of a file on a Mac, right-click on the file, then, while in the right-click menu, hold down the "OPTION" key to reveal the "Copy as Pathname" option.
-f rawvideo
this tells FFmpeg to pass the video stream as raw video data without remuxing. This step is what ensures the survival of embedded metadata versus a standard rewrap.
-c:v copy
copy the DV stream directly, without re-encoding.
/PATH/TO/output_file.dv
tells FFmpeg to use the DV wrapper for the output. To copy the full path of a file on a Mac, right-click on the file, then, while in the right-click menu, hold down the "OPTION" key to reveal the "Copy as Pathname" option.
This section taken from ffimprovisr

Transcode a Video

Transcode an MOV to MKV

ffmpeg -i /PATH/TO/input_file.MOV -c:v ffv1 -level 3 -coder 0 -pix_fmt yuv422p10 -slices 16 -slicecrc 1 -c:a pcm_s24le /PATH/TO/output_file.MKV

To run this script as a loop, first cd into the directory holding the .MOVs you want to transcode.
Then, execute the following:

#!/bin/bash
for i in *.mov;
do ffmpeg -i ${i} -c:v ffv1 -level 3 -coder 0 -pix_fmt yuv422p10 -slices 16 -slicecrc 1 -c:a pcm_s24le ${i%.*}.mkv;
done

Trim a video

Trim a video without re-encoding

ffmpeg -i /PATH/TO/input_file -ss 00:02:00 -to 00:55:00 -c copy -map 0 /PATH/TO/output_file

This command allows you to create an excerpt from a file without re-encoding the audiovisual data.

ffmpeg
starts the command
-i /PATH/TO/input_file
path, name and extension of the input file. To copy the full path of a file on a Mac, right-click on the file, then, while in the right-click menu, hold down the "OPTION" key to reveal the "Copy as Pathname" option.
-ss 00:02:00
sets in point at 00:02:00
-to 00:55:00
sets out point at 00:55:00
-c copy
use stream copy mode (no re-encoding)
-map 0
tells FFmpeg to map all streams of the input to the output.
/PATH/TO/output_file
path, name and extension of the output file. To copy the full path of a file on a Mac, right-click on the file, then, while in the right-click menu, hold down the "OPTION" key to reveal the "Copy as Pathname" option.

Variation: trim file by setting duration, by using -t instead of -to

ffmpeg -i input_file -ss 00:05:00 -t 10 -c copy output_file

-ss 00:05:00 -t 10
Beginning five minutes into the original video, this command will create a 10-second-long excerpt.
This section modified from ffimprovisr

Run a loop using VideoAIP

Creating a loop with VideoAIP

This loop will run VideoAIP on all .mkvs in a directory. Instructions for modifying to accept other extensions is presented below.

#!/bin/bash
while read TARGET
do videoaip -l auto "$TARGET" </dev/null
done < <(find /PATH/TO/DIRECTORY -name '*.mkv')

VideoAIP will try to process any files that it comes across. To ensure VideoAIP only processes the file extensions you've defined, you can replace the last line of the script with:
done < <(find /PATH/TO/DIRECTORY -name '*.mkv' -o -name '*.dv' -o -name '*.mov')

This command will find all .mkv, .dv, and .mov files in the defined directory.

Scheduling a Cron Job

Using Crontab-UI

To use crontab-ui, first install node using:
brew install node

Then, install crontab-ui using:
npm -install -g crontab-ui

You may need to use sudo when running the command above.

From the terminal, run crontab-ui

You will see the following output:
Node version: [CURRENT VERSION]
Crontab UI is running at http://[IP ADDRESS]

Now, open your web browser and navigate to http://[IP ADDRESS]

From this page, you'll be able to schedule, edit, save, and delete jobs.

Here is the format for a simple Cron Job:
0 22 * * * /Users/MIPoPS/Desktop/BackUp.sh

The first part is 0 22 * * * This is where we schedule the timer. This job will run at 10 pm every evening.
The rest of the line is the command as it would run from the command line. This job would run the BackUp shell script from MIPoPS desktop.

Each asterix represents a unit of time, as follows:
Minute(0-59) Hour(0-24) Day_of_month(1-31) Month(1-12) Day_of_week(0-6)

Use MILITARY TIME!

This cron job will run at minute zero, every hour (i.e. an hourly cron job):
0 * * * * [command]

This is also an hourly cron job but run at minute 15 instead (i.e. 00:15, 01:15, 02:15 etc.):
15 * * * * [command]

This will run once a day, at 2:30am:
30 2 * * * [command]

This will run once a month, on the second day of the month at midnight (i.e. January 2nd 12:00am, February 2nd 12:00am etc.):
0 0 2 * * [command]