Seam carving

Seam carving is a technique developed by Shai Avidan and Ariel Shamir that enables content-aware image resizing while preserving visual coherency. To scale an image, seams are inserted or removed based on a energy function that can be of many different types. Low-energy pixels that are similar to their neighbors are removed, while high-energy ones are preserved. The pixels are added or removed in real time. Except for resizing, the technique can also be used for content enhancement and object removal.

Even when it’s said not to be perfect, some of its drawbacks can be eliminated by combining it with other techniques or choosing a best approach for a particular image type. However, this means that the resizing process can’t always be done automatically and may require manual effort to select the parts of the image we want to keep. This isn’t always practical. Images with condensed content or ones with objects spanning across large areas in a way that stops the connections between seams may be also hard to resize without breaking.

One could assume that applying various different algorithms to an image will be slow, especially in the context of a website, where real-time feedback is needed. If many images need to be resized simultaneously, computing hundreds of seams may fully utilize the CPU and make the painting times unacceptable. The authors of this technique prove that resizing an image takes linear time to the number of (connected and disconnected) seams in the image. Although this can still be slow, they propose a way to pre-compute the various image sizes by encoding the data how the seams would behave during resizing (in an index map). This is a trade-off, because it makes image resizing smoother at the expense of storing more data in the image.

Nevertheless, this approach is interesting and valuable, because it concentrates on the actual value within an image, which is important in web design, where visual content is often the largest piece of the page weight pie. If an image has extra amounts of sky or greenery, it’s not always justified to download them and show them to the user. We can crop the needless areas to reduce the size, but as the authors have shown, this may also reduce the overall energy in the image. Additionally, it’s not always a good idea to scale the image proportionally when it needs to be shown on small-screen mobile devices, especially when we want the main subject to remain visible. If improved, this approach could enable the use of images of reduced size while using the encoded data as an enhancement that is lazily provided by the browser in case of viewport resizing. If we are able to cache this data in the browser, then no additional HTTP requests will be needed. It remains only to be seen whether the differences in before/after image quality will be acceptably small on different types of images. If you want to learn more about seam carving, you can read the original paper or watch this demonstration.