PANINI-VIDEO USERS GUIDE version 0.4 beta 19 Nov 2012 This guide and the software it describes are copyright 2012 by Thomas K Sharpless. Panini-Video is licensed software. All copies remain the property of the copyright holder; you are licensed to make copies for execution on your computer only. Test versions are licensed at no fee to testers who agree to the test licensing terms; these licenses expire after a fixed time. Licenses for production versions must be purchased. OVERVIEW Panini-video converts wide angle and fish eye video to natural perspective using a "software lens" based on the general Panini projection. It can transform fish-eye video into ultra-wide angle images that resemble ordinary wide angle, and correct perspective and lens distortions in video shot with wide-angle lenses. It can also correct small camera pointing errors, and simulate panning within a wider field of view. Transformed images can be of any size, shape and magnification relative to the original. Thus Panini-Video is a good tool for fitting wide angle video to different display formats. Calibration of the source camera is required; however for most purposes this is easily done by eye. Panini-Video is a file-to-file transcoder, like ffmpeg, but with a GUI and a focussed mission. Using the ffmpeg libraries (LGPL version) it can read most video file formats. It writes .MP4 files in MPEG-2 encoding, a format accepted by nearly all video editors. Default output quality is about equal to the input. Processing speed is limited by the video codecs and will be similar to ffmpeg transcoding run at the same settings. All image processing is done in the native video color space, with conversion to RGB only for display, so output color should match source color very closely. The output frame dimensions are freely adjustable. Default is same as the input, and several standard sizes are selectable. Output frame rate is same as input. Sound can be copied from many but not all source formats. MAIN WINDOW AND TOOL BARS The main window displays transformed images. It is freely resizable, but image shape always matches the selected output frame dimensions. There is a standard menu bar for "file" and "help" operations, and two tool bars: the operations toolbar, normally at the right, and the video toolbar, normally at the the bottom. You can drag the tool bars to other locations or let them float. The first button on the operations bar, that looks like a TV set, shows/hides a monitor window that displays input video as-shot. The next three operttion buttons show/hide control windows for calibrating the camera, tracking camera attitude, and adjusting the output projection. Four VCR-like buttons on the video toolbar control scanning the surce and recording output: Rewind: closes and reopens the source file Play/Pause: starts/stops rate controlled reading Step: reads one frame Record: opens/closes an output file Next to them are a counter that shows the position of the current frame in the source file, and a box that shows the requested frame rate during Play. When the main window has focus, the up and down arrow keys change the frame rate and the spacebar has the same effect as Play/Pause. ==> the actual frame rate may be limited by video decoding and encoding time. Panini processing is done on the GPU and is very fast compared to the codecs. The field of view of the output image, in degrees, is shown at the end of the video toolbar. Given correct camera calibration, these numbers are exact for rectilinear projection and fairly accurate over most of the range of the projection controls. LOADING A VIDEO FILE On the File menu, 'Load a video' brings up a file selector to locate the input file. Almost any kind of video file is OK. You can also drop a file on the program icon to launch P-V. When a source file is opened, Panini-Video puts up a "file setup" dialog. This shows the name, directory and image size of the source file, and lets you set a name, directory and image size for the output file. You can accept the automatic default settings if you like. The file setup dialog also offers these options: -- whether to rotate the source by 90 or 180 degrees. Select the source edge that should be at the top of the target frame. -- whether to output sound (if source has an audio stream). -- the name of a preset camera calibration. -- the name of a preset projection. -- an estimate of the horizontal field of view. Not required, but can help P-V select good default settings. Click OK to apply the displayed settings. Next you should set the lens correction, or check that the preset correction is OK. ==> whenever you click Rewind, P-V re-displays the file setup dialog, so you can change settings. This works even after you cancel out of the settings dialog. BASIC CAMERA CALIBRATION Before it can do its re-projection magic, Panini-Video must 'undo' the camera's projection. Like a panorama stitcher, it uses a model of the camera and lens to create an ideal spherical image, that it then re-projects with a software lens. The quality of the result depends very stongly on the accuracy of the model. Pano stitchers can start with a rough approximation, which they optimize as part of the image alignment process. But Panini-Video has no way to optimize camera parameters -- you have to give it a good calibration before it can give you a good image. Fortunately Panini-Video's camera model is easy to adjust by eye. So the first thing you should do is open a source file, find a frame that shows several straight lines and click the camera icon on the operations bar. The displayed projection will switch to rectilinear, and the calibration controls will appear. Now set the dFOV slider to the approximate diagonal field of view, in degrees, and the Curve slider to 0 for a 'normal' lens 1 for a Samyang stereographic fish-eye 2 for a classic (and rare) 'glass ball' fisheye 2.8 for a modern 'flat face' fisheye Then touch up those two controls until all the staight lines look straight. If the same settings don't straighten both horizontal and vertical lines, adjust pixAR unti they do. Having found a good setting, you may want to save it for future use. Type a name in the box in the calibration window, click save. It is not yet possible to import professionally made calibrations, but For complete details on calibrating a camera, please see Appendix A, below. ADJUSTING THE PROJECTION Panini-Video has six projection control parameters. Three of them reshape the image: Compression controls horizontal angle compression. This can vary from a 'hyper-rectilinear' edge stretch at values below zero, to extreme edge compression above 100. Zero gives the standard rectilinear projection, 100 the standard Panini projection. Compressions between 20 and 80 will be best for most images. Squeeze controls vertical angle compression. This can reduce the 'bulging' of horizontal lines in the top and bottom center areas, that is a feature of the basic Panini projection. But too much squeeze can lead to unnatural double curvature of those lines. In general you need more Squeeze at higher Compression. Height adjusts the image height linearly. Compression and Squeeze change this nonlinearly; use Height to keep the shapes of your subjects within believable limits and/or trim the aspect ratio to suit the frame. The above controls keep image size roughly constant, however they necessarily change its shape. So you may need to adjust overall size, and sometimes to shift the image sideways or up and down, to best fit it to the frame. That is the job of the other three projection controls: Zoom adjusts overall image size linearly. Hshift moves image linearly in the horizontal direction Vshift moves image linearly in the vertical direction When the reprojected image does not completely fill the frame, you will see dark green areas (with "pointy" borders due to the triangle mesh used in the projection). To get a better feel for how the controls affect shape, zoom out until you can see the whole border of the image. If you zoom in you can use the shift controls to center any part of the magnified image in the frame. The best combination of compression, squeeze and height settings will depend mainly on the field of view. In general, the wider the view, the more compression and squeeze are appropriate. Here is a very rough guide: Regime FOV, degrees Compression Squeeze wide angle 60-90 0-50 0 super-wide 90-120 20-70 0 ultra-wide 120-150 40-90 0-30 hyper-wide 150-180 60-110 20-50 ==> There is no "right" way to set the projection. Your eye is the final authority. If it looks right, it is right. Experiment freely. PRESET LENS AND PROJECTION SETTINGS The lens and projection windows can save their current settings as a "preset" whenever you like. Type a name into the box, click save. The names of all saved settings are in a drop-down list below the box. Selecting one makes it the target for Remove, Save or Load, but does not load it -- click Load to do that. In the file setup dialog you can select presets for the new file. CAMERA ATTITUDE CORRECTION The Panini projection works best when accurately aligned with the "up" direction of the real world. And small changes in camera yaw or pitch have big effects on the perspective of wide angle images. Panini-Video has controls that can correct camera roll, pitch and yaw over a limited range. The "tracker" icon, a box with cross-hairs, opens the control window. These controls rotate the ideal image before the Panini projection is applied, so the effect on image geometry is the same as if the camera were rotated around the lens nodal point. For example converging verticals, due to the camera pitching up or down, can be made parallel. Of courese, these corrections move the image relative to the frame, and change its overall shape. P-V keeps the image centered vertically when you adjust the pitch tracker, but using any of these controls can make 'green areas' show, you may have to adjust zoom, height, and/or shifts to compensate. ==> Compensating those shifts is not practical while you are trying to track a moving camera's roll and pitch. Future versions will have better automatic the compensation. And eventually the camera tracking controls will respond to attitude sensor readings recorded by pro cameras. RECORDING Clicking the record button creates a new output file, or "clip". The button is solid red when an output file is open. Then either play or single-step will send frames to the file. Clicking the record button again closes the output file (and stops playing). The rewind button will also end recording. You can pause, step, resume play and adjust play speed while recording. None of that affects the frame rate of the output video. You can also freely adjust the projection, and apply camera attitude correctons. The directory and base name for output files are set when the source is opened. Clips are named by appending -clip# to the base name. # represents the clip sequence number, which starts at 1 when the source is opened and increases by 1 for each clip recorded. ==> alpha-1 overwrites existing clips of the same name without asking for permission. ==> Alpha-1 cannot copy all sound formats. It does not know which ones will work until it tries to open an output file; so if you get the message "of_create_file failed" when you click the record button, try re-opening the source with sound disabled. KEYBOARD SHORTCUTS => These work only when the main window has focus (click anywhere in it). Unfortunately they do not work while the mouse is in a control window -- this will be corrected. -- spacebar = Play/Pause -- up/down arrows adjust requested play rate APPENDIX A -- How to Calibrate a Camera Eventually it will be possible to import professionally made camera calibrations into Panini-Video. But for the moment you will need to learn to calibrate the camera by eye. With practice and attention to detail you should soon be able to get consistently good results. When the camera calibration window is visible, Panini-Video shows a pure rectilinear projection, in which all lines that are straight in reality should look straight on the screen. If they do not, you need to adjust the calibration controls. The contols are "dFOV", "Curve", "CorrA", "CorrB", and "pixAR". The most important camera parameter is the ratio of lens focal length to sensor size, which determines the overall angle of view. As the name implies, dFOV represents the diagonal field of view, measured in degrees. For circular fish-eye lenses it is the diameter of the image circle. Curve is a number between 0 and 3 that defines the basic shape of the lens's projection curve, in terms of the 4 ideal curves that lens designers most often try to approximate: 0: rectilinear ('normal' lenses); 1: stereographic (Samyang fish-eyes); 2: equal-angle (classic 'ball' fish-eyes, Nikon and others); 3: equal-area (most modern 'flat' fish-eyes) Between whole numbers, the curve is a blend of the two nearest ideal curves. CorrA and CorrB generate a polynomial correction to the ideal curve. They are not the actual coefficients of the correction polynomial, but parameters of a function that computes those coefficients. The parameter "pixAR" represents the height to width ratio of the camera's pixels. It is needed because many video cameras have non-square pixels. Although hard to notice (and easy to compensate) in linear images, this has to be accurately known to compute the ideal image from a wide angle source. When the pixel aspect ratio is not set properly, it is impossible to make horizontal and vertical lines straight at the same time. The lens parameters are meant to be adjusted in this order: FOV, Curve, CorrA, CorrB. Each has a weaker effect than the one to its left. Indeed, the effect of CorrB is too subtle to be judged visually in any but very special images. The parameters interact somewhat, and various combinations may seem to give the same result, but with good test images and experience you will find that there really is a best combination for each lens. That combination will normally have FOV and Curve close to the values one would predict from the lens design parameters and camera dimensions. Adjust pixAR until both vertical and horizontal lines become straight at the same setting of the other controls. A good test image has lots of straight lines, at many different distances from center. If there are many truly vertical and horizontal lines, it may be helpful to adjust camera attitude so that they are accurately vertical and horizontal on screen. The goal is to make all straight lines look perfectly straight at the same time. Each control changes curvature differently for lines at different distances from center. Try not to be influenced by how nice the perspective looks, just how straight the lines are. Holding a transparent ruler against the screen can be helpful. Ignore lines that pass through the image center, because those look straight at all correction settings. And beware of lines that are not absolutely straight, such as horizontal wires. A good test subject for calibrating cameras is several weighted strings hanging from the ceiling of a large room. Make sure some long horizontal lines are also visible, such as baseboards or strings taped to a wall. The vertical strings should cover the whole field of view, ideally at equal angles on a large circle around the lens pupil. APPENDIX B -- Command line API usage: pgm_name [options] [files] [options] Each option is 2 'words' the first of which begins with 2 dashes. The 2nd may be a list separated by commas, without any spaces. --out wid,hgt,fov : set output frame --sound n : read sound from source n = 1 to number of source files; 0 no sound. --czvq cmpr[,zoom[,vscl[,vsqz]]] : projection cntrol settings (numbers, no spaces). --ulm hfov[,curve[,corrA[,corrB]]] : lens calibration settings (numbers, no spaces). --up l|t|r|b : up direction in filed image up, ulm or no option implies that source needs lens correction. For panoramic video use: --pano 'equi' or any permutation of lfrbud : read as equirectangular or as cube faces in given order. equi implies one file with 2:1 aspect ratio. cubic can be one 2:1 file, each frame a 3x2 array of square cube faces, or multiple files with square frames. In that case a partial cube is allowed: specify a subset of lfrtbud and supply the same number of files. [files] Full paths to one or more video files libav can read, or to sequences of separate jpeg, tiff or png images. If pano is given, number of files and image shape must agree. To read sequences of separate images, the name extension must be .jpg, .jpeg, .tif, .tiff, or .png, and the file name must contain an embedded format symbol representing the frame number: foo/bar%d.jpg C:/stuff/split/frame_%04d_left.tiff The simple form %d gives variable-width numbers, %04d gives 4-digit numbers with leading zeroes. It is best to use forward slashes on all systems, however backslashes may work on Windows.