Tell us about your work
In the series of ‘Jewels of KGP’, this time we bring you the one who has surely raised the flag of IIT Kharagpur to tremendous heights. He is none other than Mr. Parag Havaldar. He is honored with the Technical Achievement Award for the development of expression-based facial performance-capture technology at Sony Pictures Imageworks. He had done his B.Tech. in computer science and engineering in 1991 from IIT Kharagpur and went on to do a Ph.D. in computer vision and graphics from the University of Southern California in 1996. Currently, he is a software supervisor at Sony Pictures Imageworks, where he leads the company’s proprietary efforts in the area of performance capture.
The technology developed by him and his group have been used to create stylized and realistic character animations in a variety of movies including “Alice in Wonderland”, “Monster House”, “Hancock” and “Spiderman”.
How did your dreams and aspirations change over the period of time? When did you find yourself interested in the field of multimedia and image processing?
I am sure you agree that dreams and aspirations change over time depending on experiences you go through and people you meet and get inspired by. As a young child, I wanted to join the medical field, but some of my experiences with math and science changed that to attending an IIT. I don’t know when I got interested specifically in the visual fields of computer graphics, computer vision, images, videos and multimedia in general. I always enjoyed geometric and spatial reasoning, and luckily my undergraduate project presented an opportunity to learn a lot about processing digital images, which you never got as an undergrad, and that led me to study computer vision at the University of Southern California. The professors I met had research interests in imaging and vision – from the low level signal processing aspects to high level cognitive brain models of human and computer vision. Being in the Los Angeles area, I was at the right place for entertainment and at the right time where digital technologies were poised to make a big influence on entertainment. So you never know what opportunities come your way, who ends up inspiring you. Having aspirations and a desire to do something is the very first step, one only hopes that you are also fortunate to be surrounded by mentors to guide you and opportunities to explore.
When did the thought of receiving the “TECHNICAL OSCAR” cross your mind? How is the feeling of being on the receiving end of it?
When a large and influential organization like the Academy of Motion Pictures gives one an award such as this, for developing a technology that has influenced moviemaking, one has no answers. I feel very humbled and grateful. At the same time, I do feel overjoyed that so many years of work was recognized at this level but I also feel tremendously responsible – with a need to advise and influence younger generations in a practical way so that they make informed decisions in their scholastic and professional endeavors. That said, I don’t think you can ever work towards such an award, or for that matter any recognition. When you are engrossed in any work, you are always caught up in that moment to solve problems. An award of any kind was never in my mind. But I also need to express that some of my other work in a different area was investigated/nominated by the Academy for a potential award in 2011, only that the Academy decided that the technology was premature and not influential enough. So you never know!
What role did IIT KGP play in your overall development as an individual? Share some of your tough phases and how did you overcome those?
IIT KGP has played a very formative role for me, it has done so for all the wonderful friends I made there and it will for all of you. We leave the comfort of a home at a very pristine age, ambitious but foolhardy, to start a special phase in our young lives – that of education at KGP. Our premature ideals and inchoate goals are bound to take a sail in such an environment by sharing the same with friends that we end up making. I grew up as a kid in various places – Pune, Ahmedabad, Ndola, Lusaka, Livingstone, Nairobi, Blantyre, Arusha, Kampala. So learning different languages and peer assimilation was always a struggle. You learn very quickly that you have to be good at something to gain respect. But in a place like KGP, it is hard to be good at something – the selective nature of the process ensures smart, well read, talented individuals. My seniors were charismatic and embodied leadership qualities that we wanted to emulate. They were always a great source of inspiration. I took part in sports, cultural events and learned to speak Bengali in a few months. Comparing notes afterward with my friends from other IITs, I can conclude that IIT KGP was very different, because it was not a city and nowhere close to another big city when I embraced it. This might have changed now, but during my times most students stayed back on campus all weekends. The forced togetherness created an ecosystem for spending more time together leading to exchange of ideas, development of ideals and friendships – that played a role in sculpting our youth and consequently laying a pathway for our future. So, conclusively and decisively I can say that KGP has its own benefits.
How was your stay at the University of Southern California as a Ph.D. student? How challenging do you find being a part time faculty there now?
USC was great and in fact, my experiences at KGP prepared me well for it. After the rigor that the computer science department at KGP subjected us to, you are ready to learn more. The research exposure at USC was a breath of fresh air. Of campus life was easier to deal with in terms of roommates, friendships, cultures etc. because you had already met and dealt with all the possible personalities while growing up in KGP. The research groups and faculty with very progressive, plus the proximity to the industry that you get in a large city like Los Angeles has an added benefit and exposure. The peers I graduated with at USC have become fairly influential in their own right, including the current chairperson of the department.
Now, as a part time faculty at USC, I enjoy the experience thoroughly because it has allowed me to give back to the community with a unique perspective of having worked in the industry. It was an uncanny experience when I began teaching at USC because I became a faculty member few years after I graduated. So I had not lost my student perspective and could relate to all students, especially international students. It was a challenge to not act like a student amongst them. But over the years, you meet so many fresh and young bright green minds, who never stop challenging your own learning. That is what I love about graduate research schools– the curiosity that young smart minds bring to the environment.
Kindly share with us how it felt to lead the crew of performance capture at “SONY Pictures Imageworks”
Looking back, it was a wonderfully unique experience. The movie industry is all about creating pixels on the screen, held to a very high standard of storytelling. Technology is always great, but the movie industry is a creative endeavor and it was always important to understand the impact that computational algorithms and automation can have on a creative client base. The clients we had, especially director Robert Zemeckis, Spielberg were well aware of the technology but in the end, it does not matter if the technology does not deliver in the creation of the final art form. I led a small but very talented group of six to ten computer visionaries who had to be all conscious of not only the advantage but also the limits of technology, under production pressures. Just because your algorithm works, does not mean a movie production of an army of artists will use it if they cannot control it to get what they want. It was more about our team understanding artist workflows and artistically controlled outputs. In the end, the greatest sense of accomplishment for me was leading engineers to think about the synergy between technology and art!
How is your current work with “Blizzard Entertainment” as R&D lead going on? What advances can we expect in this field in near future?
At Blizzard, I lead an R&D group that model creates and animates digital characters, but for the gaming industry. Blizzard Entertainment, as part of Activision-Blizzard, is a big computer gaming company with some high profile multiplayer games such as World of Warcraft, Overwatch, Starcraft, Hearthstone. With faster computing platforms, such as GPUs. with dedicated gaming hardware (PlayStation, Xbox), and new interface paradigms such as Virtual Reality, computer gaming is poised to transition from what it is right now to a more character-centric and emotionally engaging game experience. Interaction with digital characters will be a big part of this change. Also with the confluence of multi-disciplinary progress in the computer sciences such as natural language processing, audio analysis, and synthesis, cognitive computational models we are bound to see intelligent digital characters. This might lead to a transformation in entertainment from a passive “story telling” experience as we witness with movies, to a possibly active and even interactive “story defining” experience. Computer gaming technologies that advance digital characters at their helm will be at the forefront of such an evolutionary entertainment paradigm. At that time, you perhaps won’t be sitting in Netaji and shouting “Tarapodo lights!”
“Alice in Wonderland”, “Spiderman”, “Hancock” and many more, all were truly a masterpiece. Which one is your personal favourite and why?
It is hard to define any one project as a favorite. The creative/artistic needs of each project are different, requiring different technological advances. My most memorable projects have been ones in which I was involved in research from the very beginning, went onto production sets and interacted with the creative side people to understand requirements of technology, or see first hand how well (or not so well) our technology worked. For instance, creating virtual actors on Beowulf with Angelina Jolie, Hancock with Will Smith was very challenging because digital doubles of actors such as Jolie and Smith need to animate and remote with immaculate accuracy on screens to sell a performance as “real” to the human eye. Automation in this regard was really really hard! Monster House was a different challenge because we had to capture child actors and transfer their actions onto digital stylized/cartoony characters creating a very nuanced animation. I also enjoyed leading a team of computer scientist and artists to help create stereo movies. The problem of creating stereoscopic (two) images captured from single (one) vantage viewpoint of the film camera (or any camera) is still not a completely solved problem even in research and academia. At SONY we have always enjoyed interacting with great visionary directors like Robert Zemeckis on Beowulf, Tim Burton on Alice in Wonderland, Sam Raimi and Marc Webb on the Spiderman stories and their projects have always needed solutions which were a privilege to address.
What suggestions would you like to give a student interested in “computer vision and graphics”?
I hope not to get too technical! Firstly, understand the difference between computer vision and computer graphics. Computer vision deals with real visual (images/video) signals acquired by cameras as input to computationally extract intelligent descriptions – metrically accurate for engineering purposes as well as analytically sound to derive models of human perception. For instance, you might reconstruct the world around you seen via photographs – to the very millimeter, or/and you could describe intelligently what your signal has sensed – who is there in the image, what are they doing, Computer graphics aims to take accurate descriptions (such as those that computer vision is trying to extract) and produce realism in rendered images – … broadly speaking both processes are the exact reverse of each other. And this entire area has greatly changed since I was inspired by that problem space as a student. It has now spawned into a variety very insightful and much needed research and industry explorations. Today, there is a lot of research on mapping the human brain, and how the human visual system works, but computational models that describe these aspects correctly are still needed. Machine learning, especially deep learning is at the confluence of solving a lot of problems today but these solutions are still very typified to solve specific problems and not unified. The field is poised to make influential progress in the coming years. Computer Graphics – a lot of progress in terms of modeling, shading, animation and rendering to make virtual worlds seem very real. But there is a problem of scale – to create content for movies, games, virtual reality still, takes a lot of human labor. There need to be algorithms, tools, and pipelines that can democratize this effort. – still largely perplexing. In short, these related fields of vision and graphics, (like all other interesting areas!) have more questions than answers. If you have intriguing and purposeful questions after initial research, I would love and welcome conversations.