Sunday, April 3, 2011

Revisiting "Playable Audio and Video in ePub" (revised 4/3/2011)

Last September, I looked at the need to make sure that video embedded in an ePub document would play and play well on all of the devices where the iBooks.app is found (iPhone, iPod touch and iPad). Since then, new developments not only make this topic more important but also warrant a review of the principles involved.

New developments include the fact that the ePub standard is now at version three and formally specifies the embedding of audio and video files in ePub documents using the same HTML 5 video and audio tags that the iBooks.app has long supported. Thus, there will likely be more eReaders supporting rich media beside iBooks. When those new eReaders do arrive, we'll need to look at them closely. For now, the iBooks.app is still the only game in town for rich media in ePub documents. As well, we have new hardware and operating systems to consider and that brings up "backward compatibility" as an increasingly important factor.

So, let's take another look at this topic and bring our earlier conclusions up to date.

Hardware Targeting The iBooks.app, now at version 1.2.1, requires iOS 4 or later so that reduces the range of hardware that it will run on to iPhone 3G, iPhone 3GS, iPhone 4, iPad plus the 2nd or later generation iPod touch. The most recent of these (iPhone 4, iPad and 4G iPod touch) sport the significantly more powerful A4 processor and, thus, are capable of decoding more advanced versions of the MPEG-4 H.264 standard. Specifically, these newer models can handle H.264 Main Profile whereas the older models can only handle the Baseline Profile. Simply put, baseline will play everywhere iBooks runs whereas video encoded using the Main Profile will only play on newer iOS devices. Thus, aiming for the greatest reach means that we need to stick with Baseline Profile for maximum backward compatibility.

Viewport Targeting Thanks to an article by video wizard Jan Ozer entitled "Encoding for the iPad", I've learned that, thanks to the video scaling function on the iPad, encoding HD (16:9) video at 640x360 pixels looks great for most content regardless of viewport size. The exception to this is video with fine detail such as smallish text. That gets very blurry but since we're talking about ePub here, that shouldn't be a problem. Save the text for the body of your eBook and use static images for fine visual detail.

Bit-rate & File Size Targeting An ePub document is self-contained. Generally speaking, there are no external dependencies for video or audio. Thus, video, audio, image and text files are all found inside the ePub file. Actually, an ePub document is a special kind of Zip archive that eReader apps know how to handle. Because of this self-sufficiency, bit-rate is not as critical as it would be if ePub video required an Internet download as a web page does. On the other hand, video file size is a function of bit-rate and, so, has a lot to do with the file size of your ePub document, how long it takes to download and how long your audience will have to wait before they can start reading. A frame size of 640x360 and a bit-rate of 1.4 Mbps (mega-bits per second) using the MPEG-4 Baseline Profile should produce pristine video for most HD (16:9) sourced content.

The Golden Mean Of course there are good use cases that diverge from this golden mean. The iPad's 1024x768 screen, for example, will accommodate 720p (1280x720) by scaling it down to 1024x576. The iPad will also support MPEG-4 Main Profile and very high bit rates for optimal quality playback. However, unlike other media synched to iOS devices via the iTunes.app, the video content of self-contained ePub documents is not evaluated, Thus, it is quite possible to create an eBook containing video or audio that will not play properly or at all. It is critical that ePub documents be tested on the smallest, least capable iOS devices as well as the iPad and iPhone 4 if your goal is to have the video in your ePub documents accessible wherever iBooks is found.

Reviewing Our Recommended Methods Previously, I had recommended using the iPhone presets that are found in the output options of many applications such as QuickTime Player, Screenflow, Miro and Handbrake. This is still a viable strategy though it is not optimal. The iPhone preset in QuickTime Player, for example, currently encodes a 480x300 video at 880 Kbps using the Baseline Profile. It will scale and play everywhere that iBooks runs but its not the best that we can do. Actually, the iPod preset comes closer at: 1.463 Mbps, 640x360, Baseline@3.0 when the source is 1280x720. This is much better for 16:9 source.

Scalability Testing As mentioned above, testing is critical. Not only do we have to test on multiple devices but we also need to test using all of the orientation and scaling options. Here is an illustrated list to guide your testing:

The iPod touch and the iPhone all share the same screen size (viewport). Here is the portrait view:



Here, we see a proportional scaling of the video constrained by the width of the screen in this orientation. Note the "expander" button in the upper right corner of the center image. Tapping that button fills the screen without regard for aspect ratio (proportion) like this:



Notice that the expander button has now become a collapse button.

Next, we look at the landscape orientation where the same controls are operative but the visual effect is, of course, different.



Next, we look at the iPad where video can be played in-situ, full-screen (proportional) and full screen (non-proportional). We start with the portrait in-situ view.



Here, the iPad advantage is that video can be played in context with other book elements such as text. Tapping on the expander button will bring us to the proportional full screen view which looks like this:



Notice that the collapse button on the video control bar (bottom) is the same in both images. Tapping this element will return to the in-situ video where you may continue watching or stop the video and resume reading text. The expander buttons at the top have a different function. The one on the leftmost image expands to the non-proportional full-screen image on the right. The one on the rightmost image returns the full screen proportional video on the left.

All of these views need to be checked in order to assure that your audience is seeing the video as you intend. Fortunately, the algorithms that Apple uses to scale video in these widely different environments is very good. If you start with high quality source at 1280x720 and then export using the iPod preset, you will likely be satisfied with all of these views.

6 comments:

  1. Frank,
    We've created an embedded video & audio ePub using Pages, simply by copying and pasting the files in. There are some limitations beyond what you state, but it's quite workable. This only runs on iOS devices, but it looks good and is comparable to what some apps are doing. Tell me what you think, it's a 4MB file at: http://goo.gl/gbhHB

    My email is kempton at bitmenu dot com if you'd like to exchange notes on this.

    Thanks, your blog is great.


    Kemp

    ReplyDelete
  2. I'm outputting video from ScreenFlow for an eBook. You suggest 1.463 Mbps with a size of 640x360. Liz Castro recommends 900 kbits/sec at a size of 480x270.

    I'm guessing that both of these videos will play back in iBooks since you are both ePub specialists. Are these data rate and image size discrepancies personal preferences or what? I haven't done enough testing with my ePub document to know which one to use. I have noticed that when I view the videos at full-screen with Liz Castro's settings the small text from my screen is jagged and difficult to read. I should try your settings but they produce almost 2x the filesize! My eBook has 30+ videos embedded so filesize is a concern.

    Thanks for your post. I understand this is a bleeding edge technology and am thrilled that people like you, Liz Castro and Cari Jansen post your thoughts online for everyone to devour. I'll hope for a reply to this comment, also!

    ReplyDelete
  3. I've come back to revisit your post because the embedded videos in my ePub don't play on a 3G iTouch. Somehow I missed this point about compressing the final videos at the "Baseline" preset.

    What are the specs of the Baseline preset? The MP4 vids that I'm using are 1024x768 which is clearly too larch for the iTouch to playback.

    ReplyDelete
  4. Other folks having this same problem:
    https://discussions.apple.com/message/12358772


    According to this Apple Tech document - http://developer.apple.com/library/ios/#technotes/tn2224/_index.html

    The following audio and video formats are supported:

    * Video: H.264 Baseline Profile Level 3.0 (iPhone/iPod Touch), Main Profile Level 3.1 (iPad 1,2)
    * Audio: HE-AAC or AAC-LC up to 48 kHz, stereo audio OR MP3 (MPEG-1 Audio Layer 3) 8 kHz to 48 kHz, stereo audio

    Note: iPhone 3G supports H.264 Baseline Profile Level 3.1. If your app runs on older iPhones, however, you should use H.264 Baseline Profile 3.0 for compatibility.


    This site lists the different models' resolutions:
    http://www.edepot.com/iphone.html#iPhone_Model_Differences

    ReplyDelete
  5. @graham Yes, using baseline profile 3 encoding is key to maintaining the broadest possible reach in iBooks. Now that EPUB v3 is out and supports audio/video embedding, it will be interesting to see how other eReaders behave.

    As for bit rate, I tried to find the highest common denominator for Apple devices precisely to avoid the jaggies that you saw at lower rates.

    ReplyDelete
  6. Thank you for posting this. Since I only own Apple devices, I can't test on anything else. What happens if an epub doc, with rich media embedded, is loaded onto a non-Apple device?

    Will the rich media simply be ignored? Or will the doc become perceived as corrupt? Any testing or advice?

    Thanks again.

    ReplyDelete