I3D 2011 Report – Part I: Keynote

Today was the first day of I3D 2011. I3D (full name: ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games) is a great little conference – it’s small enough so you can talk to most of the people attending, and the percentage of useful papers is relatively high.

I’ll split my I3D report into several smaller blog posts to ensure timely updates; this one will focus on the opening keynote.

The keynote was titled “Image-Based Rendering: A 15-Year Retrospective”, presented  by Richard Szeliski (Microsoft Research), who’s been involved with a lot of the important research in this area. He started with a retrospective of the research in this area, and then followed with a discussion of a specific application: panoramic image stitching. Early research in this area lead to QuickTime VR, and panoramic stitching is now supported in many cameras as a basic feature. Gigapixel panoramas are common – see a great example of one from Obama’s inaugural address by Gigapan (360 cities is another good panorama site). Street-level views such as Google Street View and Bing’s upcoming Street Slide feature use sequences of panoramas.

Richard then discussed the mathematical basis of image-based rendering: a 4D space of light rays (reliant on the fact that radiance is constant over a ray – in participating media this no longer holds and you have to go to a full 5D plenoptic field). Having some geometric scene information is important, it is hard to get good results with just a 4D collection of rays  (this was the main difference between the first two implementations – lumigraphs and light fields). Several variants of these were developed over the years (e.g. unstructured lumigraphs and surface light fields).

Several successful developments used lower-dimensional “2.5D” representations such as layered depth images (a “depth image” is an image with a depth value as well as a color associated with each pixel). Richard remarked that Microsoft’s Kinect has revolutionized this area by making depth cameras (which used to cost many thousands of dollars) into $150 commodities; a lot of academic research is now being done using Kinect cameras.

Image-based modeling started primarily with Paul Debevec’s “Facade” work in 1996. The idea was to augment a very simple geometric model (which back then was created manually) with “view dependent textures” that combine images from several directions to remove occluded pixels and provide additional parallax. Now this can be done automatically at a large scale – Richard showed an aerial model of the city of Graz which was automatically generated in 2009 from several airplane flyovers – it allowed for photorealistic renders with free camera motion in any direction.

Richard also discussed environment matting, as well as video-based techniques such as video rewrite, video matting, and video textures. This last paper in particular seems to me like it should be revisited for possible game applications – it’s a powerful extension of the common practice of looping a short sequence of images on a sprite or particle.

Richard next talked about cases where showing the results of image-based rendering in a less realistic or lower-fidelity way can actually be better – similar to the “uncanny valley” in human animation.

The keynote ended by summarizing the areas where image-based rendering is currently working well, and areas where more work is needed. Automatic 3D pose estimation of the camera from images is pretty robust, as well as automatic aerial and manually augmented ground-level modeling. However, some challenges remain: accurate boundaries and matting, reflections & transparency, integration with recognition algorithms, and user-generated content.

For anyone interested in more in-depth research into these topics, Richard has written a computer vision book, which is also available freely online.

Just two weeks until the SIGGRAPH General Submission deadline!

SIGGRAPH 2011 will be in Vancouver, on August 7-11, 2011. I’ve given presentations at SIGGRAPH several times; each time was a great experience where I learned a lot and met some pretty awesome graphics people from the world’s top research institutions, film production companies, and game development studios.

SIGGRAPH has several programs at which game developers can show their work; I wanted to point out that two of the most important (Talks and Dailies) have deadlines on February 18th, less than two weeks away! Fortunately submitting a proposal to one of these programs doesn’t take much time. However, getting approval from your boss may take a while, so you don’t want to wait.

SIGGRAPH Talks are 20-minute long presentations which typically contain “nuggets” of novel film or game production tech. These can be rendering or shading techniques, tools for artists, enhancements done to support a tricky character design, etc. If it’s something a programmer or technical artist is proud of having done and it’s at least tangentially graphics-related, chances are it would make a good Talk submission. Submitting a talk only requires creating a one-page abstract; if the talk is accepted, you have until August to make 20 minutes worth of slides – not too bad. To get an idea of the level of detail expected in the abstract, and of the variety of possible talks, here are some film and game Talk abstracts from 2009 and 2010: Houdini in a Games Pipeline, Spore API: Accessing a Unique Database of Player Creativity, Radially-Symmetric Reflection Maps, Underground Cave Sequence for Land of the Lost, Hatching an Imaginary Bird, Fast Furry Ray Gathering, and NPR Gabor Noise for Coherent Stylization. If you are reading this, please consider submitted the coolest thing you’ve done last year as a Talk; the small time investment will repay itself many times over.

SIGGRAPH Dailies are relatively new (first introduced at SIGGRAPH 2010). These are very short (under two minutes!) presentations of individual art assets, such as models, animations, particle effects, shaders, etc. Unlike the rest of SIGGRAPH which emphasizes novel techniques, Dailies emphasize excellence in the result. Every good game or movie has many individual bits of excellence, each the result of an artist’s talent, imagination and sweat. These are often overlooked, or unknown outside the studio; Dailies aim to correct that. Dailies submissions are even easier than Talk submissions. All that is required is a short (60-90 second) video of the art asset, no audio, just something simple like an animation loop or model turntable. You will also need a short backstory; something that gives a feeling for the effort that went into the work, including any notable production frustrations, unlikely inspirations, sudden strokes of genius, etc. Don’t write too much – it should take about as long to say as the video length (60-90 seconds). To get a better idea of what a Dailies presentation looks like, here are two examples. The list of Dailies presented at SIGGRAPH 2010 can be found here: it runs the gamut from Pixar and Disney movies to student projects. I suspect not many artists read this blog, so any game programmers reading this, please forward it to the artists at your studio.

Good luck with your submissions!

NPAR Call For Papers

The ACM Non-Photorealistic Animation and Rendering Symposium has put out its call for papers. NPAR alternates its location each year between Annency, France (which is lovely) or colocated with SIGGRAPH (which is convenient for many of us). This is a SIGGRAPH year, in Vancouver (lovely and convenient). NPAR takes place the weekend before, colocated with Sketch-Based Interfaces and Modelling (SBIM) and (new this year) Computational Aesthetics (CAe).

How to Make an Ebook

Here’s a short guide on creating decent ebooks from scans using Adobe Acrobat. This will not be of interest to 98% of you, but I want to record it somewhere for those of you who may do this in the future. It is written by Iliyan Georgiev, who made the recent PoDIS ebook. Comments are welcome, as usual.

The one piece of software you’ll need that can’t be downloaded for free is Adobe Acrobat, though even this application has a 30-day free trial.

1. Scan the pages of the book using a scanner (a digital camera is a good alternative).

2. Crop the scanned images (and split the pages, if you scanned two pages at once). It’s better for an ebook to have smaller page margins. Also, cropping removes black areas and other artifacts resulting from scanning. An excellent (JPEG-only) batch cropping tool for Windows is JPEGCrops. It has some disadvantages, however, so in practice it’s best to use JPEGCrops to estimate approximate cropping parameters (width, height, x-offset, y-offset) and XnView‘s batch processing mode for the actual cropping. Both applications are free and have portable versions.

3. Assemble all images into a PDF file. Adobe Acrobat has an option to combine multiple files into a single PDF. Use the highest quality settings for the creation.

4. (OPTIONAL) Rearrange/merge/delete pages. Acrobat has excellent tools to achieve these. This can be useful for books that are published in two volumes or for extending the book with additional information, such as errata listings, images, high quality cover pages, etc.

5. Manage blank pages. It might be tempting to delete blank pages inside the book. Such pages are always intentionally left blank by the publishers, as they are important for the printing order. This is particularly important for the first few pages, as well as for the chapters. Many books are created in such a way that all chapters start on an even/odd page, and the large majority have the inner pages typeset for being printed on a specific side (left/right). If you want to optimize the page count anyway, keep in mind how the book would appear when printed out (also using “2 pages per sheet” printing).

6. Number the pages. This is an often-overlooked, but very useful, option. Apart from the default page numbering, the PDF format supports logical page numbering. This can be used to synchronize the PDF page numbers with the actual book page numbers. This is very easy to do in Acrobat and should always be done. To do this, select the necessary pages, right click on them and choose “Number Pages…”.

7. Run OCR (optical character recognition) on the PDF. This is an extremely easy way to make your scanned pages searchable and the text copy/paste-able. Acrobat has a good and easy to use built-in OCR tool. You will find it in the Document menu (Tools pane in Acrobat X). Be sure to disable image resampling, as by default OCR will resample the images, which can easily increase the file size by a huge amount! Keep in mind that OCR is a compute-intensive process and can easily take a couple of hours for a larger book.

8. Optimize document. Acrobat has an option to optimize scanned documents. This runs some image-processing algorithms on the scanned images and compresses them aggressively when it detects text. This is a vital step to keep the size of the document low. It can reduce the file size by a factor of 20! It will also make the antialiasing to look better when pages are minified, if the resolution of the original scans is high enough. This process is also compute-intensive and can easily take an hour for a larger book.

9. (OPTIONAL) Reduce the file size further by using Acrobat’s other optimization options, from which the image downsampling is the most important.

At this point the most important steps are done and you can end here and go to sleep if you see the sunrise through the window. Go on if it’s only 4 AM.

10. (OPTIONAL) Setting the initial view. Open the document properties on the Initial View tab. Here, you can set the initial page, zoom level and which panes (e.g. the bookmarks pane, see below) should be active when the document is opened.

11. (OPTIONAL) Create a PDF table of contents (TOC). The PDF format has a useful (hierarchical) bookmarking feature with a dedicated Bookmarks pane which exists also in Adobe Reader. This feature can be used to reconstruct the book’s TOC for easy document navigation. One simple way to achieve this is the following:
11.a Go to the book’s Contents page, select the chapter title’s text and hit CTRL+B (or right click and choose to add a bookmark from the context menu). Repeat this for each chapter.
11.b Structure the created bookmarks. Rearrange the bookmarks to follow the order and structure of the book’s TOC.
11.c Link the bookmarks to pages. To do this, go over all pages of the book sequentially and every time a new chapter starts, right click on the corresponding bookmark and set the destination to the current page.

12. (OPTIONAL) Create hyperlinks inside the document. The PDF format also supports hyperlinks which can perform actions (e.g. jump to a page or a web site) when clicked. Links can be either rectangles (drawn with a corresponding tool) or text. To create text links, select the text, right click on it and choose to crate a link. There are options to set the link’s appearance and behavior.

You’re done! You have the perfect ebook and you’re late for work!

Thanks, Iliyan!

In my post about PoDIS now being free, I asked for a volunteer to reformat the two PDFs into a nicer single file. Iliyan Georgiev, a PhD student at the University of Saarland, stepped up and took over. After a few iterations, with Andrew and me providing feedback, he’s made an excellent new version of “Principles of Digital Image Synthesis”. Download this new version from here (right-click and “Save link as…”).

This new version is much superior to the one on Google Books. It’s a single, searchable file (the Google Books version is just a scan without any OCR), it has a hyperlinked table of contents (actually, two of them), and has all errata folded into the text.

Along the way I learned a new trick from Iliyan: you can put PDFs (or any file type at all) into your Google Docs area and make them entirely public, which is how this version of the book is distributed. This is good news for all researchers: you can now host your publications in your own account. No hassling with the I.T. guys to get an account, no fuss if you move to another institution. Now no researcher needs to be held back by “it’s not company policy to give public web space” or other bureaucracy. Having a web site is nicer, but hosting in this way is free and simple. You can host up to a gigabyte for free.

How to Put a Book on Google Books

I asked Andrew Glassner to outline the process he went through to free up his book from his publisher and put it on Google Books. Here’s his reply. I hope this information will encourage anyone else who has authored a book that’s now out-of-print to spend a bit of time and effort to get it out to us all.

If you plan to release your book through Google Books, the most important thing is that you own and control the copyright. Most book publishing contracts state that when the book goes “out of print,” the rights revert to the author. This is usually not automatic: you have to ask the publisher for the rights, and they have to explicitly return them to you. I usually ask for a real, paper letter with a real, human signature on it that states the rights have been returned to me (I don’t know if an email version would carry the same official weight). This is a good time to ask them for any other physical or electronic documents they have for your book, from illustrations to PDFs and so on. They’re usually under no obligation to give you these, but often they’ll give you what they have.

You’ll also need control of at least some “Territorial Rights,” which are discussed below. It’s probably easiest to get the publisher to revert the copyright and territorial rights at the same time.

Note that it’s usually up to the publisher to determine if a book is “out of print” or not. Even if it’s many years old, and they’re not putting ink on paper any more, they may still formally consider the book to be in print. I suggest contacting your publisher and first inquiring if it’s out of print. If they say it isn’t, but you think it should be, ask them why. They might just not have gotten around to giving it that status. Be polite and professional and work with your publisher to establish the status of the book as clearly as possible. When it is out of print, ask to have copyright reverted back to you. If the publisher wants to retain some rights, offer an agreement whereby they revert the rights to you, but you then assign some rights back to them. For example, you might grant them non-exclusive electronic rights, so they can provide the book on their website.

Now that you own the copyright, create an account at the Google books partner program. If you already have a Google account (say with Gmail) you may be able to simply use that; I chose to create a new account. I don’t recall if Google asks you for payment information at this point or not. If you end up charging money for any of your books, Google has to report that income to the IRS (if you’re from outside the US, I don’t know how this works). Although I was planning to release my book for free, I set up my partnership through my one-man LLC. I don’t recall if I had to give them any kind of tax information (e.g., a Federal Taxpayer ID, which is the business equivalent of a Social Security Number) at this point or not.

Once your account is created, choose “Books”. Go to “Add Books” and enter your ISBN. Often, this is the bar code printed on the book. Another source is Amazon, which usually lists the ISBN on the book’s page. Some ISBNs are 10 digits, some are 13. Try all the versions you can find until you get one that Google recognizes as your book.

Click the question mark next to Territorial Rights, look over the options, and enter the rights that you control. Note that the publisher must explicitly relinquish those rights when they return the copyright to you. I don’t know the mechanics of this step, but Google must be checking with someone, somewhere to confirm that you own the rights. When I initially listed my book, I chose “all” for this field, and a few days later Google sent me a nice email stating I didn’t own these rights. I contacted my publisher and explained I was releasing the book through Google Books and needed to control the Territorial Rights (in addition to the copyright they’d already returned to me). They were very nice about it, and a few days later let me know it was done. I don’t know what they did, but it worked.

Google will now ask if you have a PDF or physical book you can send them. I said I did not. They then said that they would let me know if and when they scanned the book in the future. They seem to say this even if they’ve already scanned the entire book. My understanding is that if they’ve scanned any of it, they’ve scanned all of it, even if they’re only displaying a few pages. So I figured that if I gave them some time, their database of scanned-in books would catch up with this request to scan in the book, and the full PDF would appear. That indeed happened about a week later.

At that point your book’s status will go to “Live”. At the far left of your book’s listing (just to the left of the ISBN) there’s a little pencil icon. Click on that. You can now control how much of the book is shown to viewers by choosing a level from the “Book Browsable” drop-down. If you choose 100%, then they will give you the chance to apply one of several different Creative Commons licenses. There’s a nice summary of them right there on the page. I chose “Attribution-Noncommercial,” so that I get credit for my work, nobody can re-sell it for profit, but other people can build upon it.

Let Google digest and process these changes.  They say it can take up to a few days.  Then your book should be ready to share with the world!

“Principles of Digital Image Synthesis” now free for download

Andrew Glassner released his two-volume book “Principles of Digital Image Synthesis” to be free for download from Google Books. This book is pretty amazing in its scope and depth. Published in 1995 by Morgan-Kaufmann Press, it provides an education in almost all the key scientific and mathematical concepts used in rendering. The human vision system and color, display systems (pre-LCD, though), signal processing, sampling and reconstruction, Monte Carlo integration, energy transport, BRDFs, and much more, in 1600 pages. I turned to it for some bits of theory for our first edition. Despite its age, it is a worthwhile volume, as the underlying science and math are still valid.

Update: get a nicer version of the PDF version of the book from Iliyan’s site, or from here. Read more about it here.

Unfortunately, Google Books doesn’t quite list the book correctly and makes it hard to find both volumes when searching. So, here are the direct links:

Just hit the PDF download link in the upper right corner of each page, prove you’re not a computer, and you’ll then have each volume. You’ll want to rename the PDFs, as Google Books calls each volume the same name, Principles_of_digital_image_synthesis.pdf. Volume 1’s PDF is 12.0 MB, Volume 2’s is 17.8 MB.

Finally, to get the errata for the book, go to the author’s page about the book.

Special bonus project: I just asked Andrew Glassner if it would be OK for someone with Acrobat or other editor to put both PDFs into a single PDF, and to fold in the errata. He said that would be great, and that he could provide a bit more new errata which is not on his webpage yet. Let me know if you’re interested in doing a good deed for the graphics community and I’ll coordinate any efforts. Update: looks like we’ve got a volunteer, and so I’m hoping a new PDF version will be available in a few days.

Tools, tools, tools

Last month I mentioned gDEBugger being free and the joys of cppcheck. Here are some others that have crossed my path for one reason or another. Please do let me know (and so let us all know) about any worthwhile tools and libraries I haven’t blogged about – part of the reason for putting out this list is in hopes of learning of tools I haven’t heard of yet.

  • There is now a free version of AQTime, a commercial application that finds memory leaks and performance bottlenecks.
  • The Intel Graphics Performance Analyzers are supposed to be good stuff, and free – you just sign up for the Visual Adrenaline Program. I haven’t used them, but know people that have (hey, there’s Dan Baker on Intel’s page – nice).
  • Intel’s Parallel Inspector, despite its name, is particularly strong at finding memory leaks in any programs. Free month trial.
  • NVIDIA’s Parallel Nsight, also despite its name and focus of its advertising, is not just for CUDA and DirectCompute debugging and analysis, it also works on DirectX 10 and 11 shaders – you’ll need two machines networked together, one to run the shader and the other to control it. The Standard version is free, though when you sign up for it you also get a time-limited “we hope you get hooked” Professional license. Due to a currently-goofy pair of machines in my office (on different networks, and one’s a Mac I use purely as a Windows box), I haven’t gotten to try it out yet, but the demos look pretty great.
  • The Windows Performance Analysis Tools are evidently worthwhile for checking coarse-grained performance and bottlenecks for Windows programs. Again, free. I’ve heard that a number of groups have used xperf to good effect.
  • On an entirely different subject, HLSL2GLSL does a good job of translating most DirectX 9 (only) HLSL shaders to – wait for it – GLSL. Open source, and more info here, which discusses related efforts (like Mojoshader) and translation in the other direction.
  • Not really a tool per se, but still cool to see: here’s a way to find out how much free GPU memory is left for your OpenGL application. Anyone know any way to do this sort of thing with DirectX and Vista/Windows 7?
  • Will WebGL take off? Beats me, but it’s nice to see there’s an inspector, similar to gDEBugger and PIX.
  • GLM is a C++ math library particularly well-suited for use with (but not at all dependent on) OpenGL.
  • Humus points out that the old workhorse PIX now has new functionality that lets you assign names to objects, making debugging easier.
  • While I was messing with his binvox and viewvox programs, Patrick Min pointed out there’s a free 3DS file format library out there, lib3ds. I tried it out and it did the job well, taking very little time for me to integrate into my own private copy of binvox.

I3D 2011 Details

Ke-Sen’s I3D 2011 papers page now has the full list. At the moment there are only 8 author preprints available out of 24 papers, but I’m sure more will appear soon. Some of the paper titles look very intriguing – I’ll write a followup blog post about them soon.

In addition, the I3D conference registration page is now up. Early registration prices range from $200-550, depending on whether you are a student, ACM member, etc. Judging from previous years, the quality of the conference is likely to be well worth the cost of attending, especially if you live in the San Francisco Bay Area and don’t have to worry about airfare and hotels.

The conference registration page also has details on hotel registration – the conference is at the Marriott Fisherman’s Wharf in San Francisco and there is a discount rate ($129 per night, only guaranteed until Jan 21, 2011). There are two ways to book a room with the discount rate:

  1. By phone: call 1-800-525-0956 and ask for the ACM Group rate.
  2. Online: go to the hotel registration webpage, enter your arrival/departure dates, Marriott reward number (if applicable), and then one of the following three codes in the ‘Group code’ option: ASSASSA (for single/double occupancy), ASSASSB (for triple occupancy), or ASSASSC (for quadruple occupancy).