Joining and Recoding Video Files


Introduction

 

Now we're onto the second version of this article. Hopefully it will seem a bit more cleaned up, and easy to follow. One of my goals is to try to explain things a little better than I've seen done around the net. Most of the guides I've found are really quite good, and all of us owe our knowledge to these guides. However, their major downside is that they drift between lay and jargon, making it confusing for the reader to understand and follow the content. I found you really had to build your understanding based on inferences made throughout the various guides. It's like the early days of Linux all over again.

 

I embarked on this journey for two reasons: 1) to rip/convert files for a home media center platform for distribution around the home network, and 2) to convert/join the good dozen or so VCD movies I got from somewhere, and hadn't yet watched. This guide focuses on #2.

 

I couldn't tolerate the idea of switching disks in the middle of a movie (laserdisc never appealed to me for the same reason). So took the VCD discs and ripped them as ISO or BIN/CUE and stored them on a Hard Drive, where they also sat unwatched for the same reason - as ISO files, you still have to reconfigure Daemon Tools in the middle of the movie. You could have created two drive letters and mapped each one, and loaded the MPEG file from each disc into a playlist - but what a hassle just to watch the movie uninterrupted! And it still wasn't perfect, because each VCD has a slight overlap between discs. Last, I could have extracted the MPEG files and delete the ISO's, but with so many on the Hard Drive to be done, I saw "big effort", and decided that if I was going to spend the time to do anything, I may as well figure out how to join them.

 

Enter the bittorrent era. I could re-download these movies. Possibly in a single file and possibly at higher quality. However, I already have them, and they're not keepers. I buy the DVD of movies I like, and these movies aren't worth downloading again. Besides, some of the bittorrents are actually split AVI files, and I also hunger for the knowledge for how to do this stuff.

 

Once I figured out and completed my first join and recode, I enjoyed the fruits of my labours and rested on my laurels. Long enough to forget how to do the job. When I sat about doing it again, I couldn't remember any of it, and decided to create step by step instructions so I would never forget again. As a result, this guide was born. I hope you find it useful and easy to follow and understand. You are welcome to [mailto:worlok+at+gmail+dot+com?subject=A+comment+about+your+BOINK+article%3A+Joining+and+Recoding+Video+Files|drop me a line] for comments, or suggestions on how to improve the guide. Yes, I know it is a wiki, and that I should open it up for people to edit. I'm afraid I don't know enough about wiki to be comfortable with that just yet. Anyway, I digress - enjoy!

 

 

Tool Set

 

You have to use alot of tools because no one tool does everything, or has all the support you may need for different video formats. Things would be much easier if two of the more popular formats we encounter (MP3 audio and MPEG2 video) were open formats that were freely available on the net. Alas, they are not, and many tools do not fully support them. On top of that, or I guess it's technically "underneath", the "layer 2" formats (DVD, VCD) each have their own requirements that require tools with a completely different set of underlying functionality than mere encoding. The bottom line is, don't go looking for one-size-fits-all tools. They don't exist. There may be alternatives to each tool, and each alternative will have its pros and cons. But to follow a path and get from A to B on your journey, you will need to encounter and make use of several tools. I will do my best to explain why you need a specific tool for a specific job, but where I fall down in this regard, please just take my word for it, and send me a suggestion to beef up the explanation a bit.

 

Tools you will need:

 

 

 

  1. You need to start with a source file, which will be an MPG file containing an MPEG2 stream (if your source is a SVCD or DVD). This article will not, at this time, cover joining AVI files. Nor will it cover working with files from a DVD (which likely don't need to be joined the way that we're going to join anyway). Nor will it cover the MPEG1 format files found on a standard VCD, though that process will likely be the most similar (I don't have any standard MPEG1 VCD's and therefore can't help you with them).
    1. Mount any ISO or CUE/BIN files with Daemon Tools.
    2. Copy out the MPG files from the MPEG2 directory to a single directory on your Hard Drive. Rename them to avoid overwrites.
  2. You will need to know some of the parameters of the video later on. I like to use Gspot, as it does a great job of this. Open your MPG file in Gspot, and make a note (yes, use pen and paper you lazy bugger) of at least the following information:
    1. Audio codec and info (will be MPEG-1 Layer 2 for SVCD sources)
    2. Video codec and info (will be MPEG-2 for SVCD sources). You need this to know what kind of files you're dealing with, and therefore which tools and plugins you need. This guide is fully assuming MPEG-2 sources from an SVCD disc
    3. "Len", which is the playing-length of the file. Useful when loading the file into your media player (VLC) later, to cue to where you expect your join to be, to see if you can visually detect it
    4. "Frms", which is the number of frames in the file. Useful when loading the file into VirtualDub later, for finding the exact frame numbers to cut/splice the join at
    5. I/L and 3:2 flag indicators, which is useful to determine how to de-interlace the video.
    6. PAL/NTSC flag indicators, and Frames/s, which is useful to know for future re-encoding
    7. "DAR", which you need to stretch the video to it's proper viewable resolution
  3. Now we need to create D2V files for each MPG file. A D2V file is an index of the MPG file. Actually I think it's really an index of the MPEG-2 stream, but I'm not the expert on that. You may or may not need this for MPEG-1 format MPG files. You need it because you need to load the MPG file into AVIsynth, which requires it. Remember I was talking about tool limitations above? Where here is your first one.
    1. Some notes:
      1. Don't load all of the MPG files at once into DGIndex, even though it supports it. You will get one D2V file, already joined end-to-end. You CAN do this, but I DON'T, because I want to leave room for the possibility of playing with the individual, un-joined files within AVIsynth. So you could say that here is your first example of an alternative way to do the same job. Okay, screw it - I'll try to document it both ways.
      2. DGIndex can do cropping, and de-interlacing. I use it for this. You can, or you can do it in AVIsynth. I have never played with de-interlacing in AVIsynth, so I will not attempt to. I have used AVIsynth for cropping though. I'm not going to explain those options.
    2. Steps
      1. Load one or more of the MPG files into DGindex. Load only one if you want to play with them individually in AVIsynth. Load them all if you don't. Go to File > Open, and find your MPG file. If you are loading more than one file, make sure you are adding them in order.
      2. Cue up (find) a frame in DGindex that allows you to clearly see the letterboxing. Letterboxing is normal on VCD's, because VCD's use a storage format of 480x480 on disc. In this screenshot, you can see the letterboxing very clearly on my 480x480 source, because the picture contains some brightness at the top and bottom. You can also see I am on File 2/3 (look at the titlebar). This is because I've loaded all 3 MPG files into DGIndex.
        1. A side note: If for some reason, your MPG files are not letterboxed the same, well, first I have to ask why, but second I have to point out that you cannot join files that are different. Different heights and whatnot - well, this just doesn't work. If you truly want to join files that are different, better to do the joining in AVIsynth as a last step, after applying a resize filter, and after making all the framerates and other properties the same. If it's only the cropping that is different, then follow this guide but do each file seperately in DGIndex. The top crop values can be different on each file, and the bottom crop values can be different on each file, but the result has to be that the cropped files have the same size as each other at the end, to allow for joining.

      3. Why crop? Well, as most SVCD's are stored at 480x480, they will be stored with the black bars at the top and bottom. They are stored letterboxed. Those black bars are "recorded" and encoded into the video. They take up storage space (not much). They ruin viewing your video in fullscreen, if your screen can display the width of the video (I'm not going to try to explain this in detail at this point, other than to say - try watching any VCD in fullscreen mode on any widescreen compter, and you will see black bars on the right and left - this is thanks to the letterboxing hitting the top and bottom your your screen). Go to Video > Cropping Filter, and play with the values and observe the result in the video display. As you can see in the screenshot, the top and bottom values might be a bit different. This is life. Click the X in the top right corner to close the Cropping window. (If you want to do your cropping in AVIsynth instead, simply write down the values for later use in AVIsynth, and re-disable the cropping filter in DGindex)

      4. Most likely, your video is interlaced. In fact, it is likely interlaced due to 3:2 pulldown. I will not explain these technically because they are explained very well "out there" (i'll provide links eventually). This is one of the areas that, although explained very well technically on the net, has a very poor or non-existent rationale explained in most guides. Most guides will tell you that this is something you need to worry about, but not explain how to deal with it or why. Let me take my crack at it: It is highly unlikely that you want an interlaced video. It is highly likely that your video is interlaced though. It is also highly likely that if you don't pay attention to the interlacing, you might not like your results. So it's simple to deal with:
        1. Look in Gspot to determine whether or not your source is interlaced
        2. Configure DGindex to get rid of it
      5. Could that be the best explanation on the net? Yes, I think it might be. So let's begin:
        1. Look in Gspot at the interlacing indicators:

        2. You can see that "I/L" is lit up. This means the source is interlaced. Thus, you want to get rid of it
        3. You can see that "3:2"is lit up. This means that the 3:2 pulldown is enabled, and is the cause of your interlacing.
        4. You can see that "Pics/s" is 23.976 and "Frames/s" is 29.970. This means that the Interlacing is caused by 3:2 pulldown to force 23.976 fps material (ie: film) to be displayed at 29.970 fps (ie: NTSC)
        5. From the DGIndex manual: "Force Film - This option is intended for the special case of film content encoded at 23.976 fps but pulled down for display at 29.97 fps, i.e., 3:2 pulldown. This option will restore the 23.976 fps film rate while keeping audio and video in sync throughout the clip. This option works by ignoring pulldown flags and inserting/removing frames to maintain a constant and sync'ed output stream. Do not use this option on non-3:2 pulled-down material, and, therefore, never use it on PAL or MPEG1. Also do not use this option on streams containing frame repeats."
        6. Go to Video > Field Operation, and enable "Force Film". This will get rid of interlacing, get rid of 3:2 pulldown, and put your output video back to 23.976 fps.
        7. Go to File > Save Project. You will end up with a D2V file containing the index for the entire movie, and a de-muxed audio stream. This process is reasonably fast. I get about 15 mins of movie indexed per minute. So most movies should take less than 10 minutes to index.
        8. Write down the vertical resolution of your cropped video. You can see it in the information window during the "Save Project" process, or you can just do the math (480 or uncropped vertical size) - (top crop) - (bottom crop) = (new vertical resolution). In my case, I wrote down "268".

  4. You're done with DGindex. Now you want to create an AVIsynth file to re-mux the audio, resize the video, and split/join at the join points to get rid of any overlap. AVIsynth files are just text files that have the "AVS" extension.
    1. Load the Audio and Video, and merge (MUX) them together
      1. Create an AVIsynth AVS file in a text editor
      2. Load the necessary plugins to read a DirectShow source, and MPA audio. Your file paths may vary. Note that one of them comes from DGindex (DGmpgDEC), and the other from VirtualDub. I can't remember whether I had to download either of these independently, or if they came with their respective programs. Sorry  The DGDecode.dll comes with DGIndex and can be found in its program folder.  mpasource.dll must be downloaded: 
        LoadPlugin("C:\UserApps\DGindex\DGDecode.dll")

        LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\mpasource.dll")
      3. Add the audio and video sources. We're not doing anything with them yet, just loading them and giving them nice, friendly names to remember them by. Note that I have included "commented out" lines as examples for those of you joining your files in AVIsynth. Since I joined my files in the indexing process (simply by generating the single D2V file from multiple input files in DGindex), I'm only loading a single video source and a single audio source 
        videoall = MPEG2Source("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-all.d2v")

        #video1 = MPEG2Source("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-d1.d2v")

        #video2 = MPEG2Source("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-d2.d2v")

        #video3 = MPEG2Source("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-d3.d2v")

        audioall = MPASource("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-all Tc0 L2 2ch 44.1 224 DELAY 0ms.mp2")

        #audio1 = MPASource("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-Chakrafile1.mp2")

        #audio2 = MPASource("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-Chakrafile2.mp2")

        #audio3 = MPASource("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-Chakrafile3.mp2")
      4. Mux, mux, mux. Again, the commented version is how you would mux the above different independent inputs together, had you not already done so in DGindex. 
        output = AudioDub(videoall,audioall)
    2. Resize. You need to change the display aspect ratio. Guaranteed that the VCD had a compressed SAR. This means that the width was not right. The width was stored at 480 because that's what it needs to be for VCD. But it was compressed using (most likely) a 4:3 ratio. to squish it into 480 pixels across. It was only compressed horizontally, not vertically. If it was compressed horizontally as well as vertically, then the maximum vertical resolution of the viewable picture frame would likely be something in the neighbourhood of 200 pixels (480, less roughly 104 top and bottom for letterboxing, leaves you with roughly 272 pixels, divided by the 4:3 aspect ratio of 1.333). So don't touch the vertical resolution, but stretch that horizontal resolution back to what it should be using the BilinearResize function in your AVS
      1. Use Gspot to figure out what your proper DAR is. It will say, right next to "dar". Convenient huh? Multiply the decimal value here by the width of your file - in the VCD's case, 1.333 x 480. This gets you 640. This is your DAR, otherwise known as the Display Aspect Ratio, otherwise known as the width of your video as you will see it.
      2. Use BilinearResize to set the width and height of your video file. Set the width to the DAR, and height to what you cropped it to, as we wrote down earlier (268, in my example). 
        BilinearResize (output,640,268)
    3. You now have a file that looks like this (I've taken out all the individual file examples). Save it: 
      LoadPlugin("C:\UserApps\DGindex\DGDecode.dll")

      LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\mpasource.dll")

      videoall = MPEG2Source("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-all.d2v")

      audioall = MPASource("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-all Tc0 L2 2ch 44.1 224 DELAY 0ms.mp2")

      output = AudioDub(videoall,audioall)

      BilinearResize (output,640,268)
    4. Load the file into VirtualDub so that we can find the join between the two clips and any repeated frames. We will cut out any repeated frames using AVIsynth, but find them in VirtualDub. VirtualDub has a really nice frame-accurate viewer, so it's easy to find a frame, and make a note of the exact frame number.
      1. With the AVS file loaded in VirtualDub, find the frames where the first clip ends and the second clip begins. Refer to the notes you took when viewing the file details in Gspot. You should have noted the number of frames in the first clip, so simply go to that frame, and you should be right next to the join
      2. Use the right and left arrow keys to frame-advance forward and backward through the join between the two clips. If the two clips overlap, some frames should repeat. You might notice the same few frames repeated, or maybe just one frame repeated a few times. You might also notice that there is no repeat at all, in which case there is no need for you to trim anything. Regardless, you want to get rid of the repeated frames, and keep the non-repeated frames. A keen eye is required here - there is no such thing as an automatic frame-repeat-detector.
      3. Write down the last frame number from the first clip that does not repeat (we're going to use frame 69995 in our example)
      4. Write down the first frame number after the join that is not a repeated frame (we're going to use frame 70000 in our example)
      5. All the frames in between will be cut out. In our example, frames 69996 - 69999 inclusive will all be thrown out. The frame numbers you wrote down will not be cut out.
      6. Go back to your AVS file, and enter the following Trim function after the AudioDub, but before the BilinearResize using the numbers you wrote down above. 
        output2 = Trim(output,0,69995) + Trim(output,70000,0)
      7. Change the BilinearResize function to resize the newly-trimmed output 
        BilinearResize(output2,640,268)
    5. Cool. Now you are done with your AVS file, and it looks like this. Save it. 
      LoadPlugin("C:\UserApps\DGindex\DGDecode.dll")

      LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\mpasource.dll")

      videoall = MPEG2Source("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-all.d2v")

      audioall = MPASource("C:\TEMP\Passion.Of.The.Christ.WS.DVDRip.SVCD-ChakraAVSEQ01-all Tc0 L2 2ch 44.1 224 DELAY 0ms.mp2")

      output = AudioDub(videoall,audioall)

      output2 = Trim(output,0,69995) + Trim(output,70000,0)

      BilinearResize(output2,640,268)
  5. Now you want to encode. Don't use VirtualDub for this unless you have some better MP3 codecs installed. Sorry, but I don't know where to get them. I just use various FFmpeg-based front-ends. AutoGK is great, but doesn't read AVS. I'm very disappointed in this. Fortunately its big brother, GK (Gordian Knot) does. So does Avanti-GUI, and SUPER. I usually use Avanti-GUI. But use whatever you're most comfortable with. In Avanti, I use mp3 (Lib) and XviD (Lib), bitrate of 1200, and all the Destination Video dropdowns to "Source". You might want to limit your encode duration to 3 or 4 minutes for the first go, just to verify the result is as you expect.