Future innovations in healthcare, manufacturing, aerospace and other critical industries will be driven by pioneering advancements at the intersection of Augmented Reality and Artificial Intelligence.
Ankur Gupta, our next pathbreaker, works as Principal Researcher at Magic Leap (Zurich, Switzerland), a company that not only makes Augmented Reality accessible but is also redefining the way people think about AR.
Ankur talks to Shyam Krishnamurthy from The Interview Portal about his fascinating journey and defining experiences in the fields of Computer Vision, Medical Robotics and everything coming together with Augmented Reality !
For students, your journey should be defined by a commitment to long-term goals over short-term gains !
Ankur, tell us what were your growing up years like?
I was born in a Tier-3 city in the northern state of Punjab, where I spent the first 10 years growing up and assimilating my core values from the environment around me. Observing my father, who is a civil engineer by training, is one of the first memories I have of myself keenly observing a “process”. His thought process was, and still is, highly compartmentalized, specializing in breaking down a problem into smaller sub-problems and solving each one of them meticulously.
Surrounded by accomplished family members, including my mother who is a postgraduate in chemistry, with a keen interest in medicine, I was fortunate to grow up in an environment that valued education and rational thinking. I was therefore encouraged to assess myself routinely and find “patterns” in things I did well. Despite modest financial means, the emphasis on learning was paramount.
During my childhood, I pursued sports competitively, received professional training in music, and enjoyed strong connections with my extended family. These early experiences played a pivotal role in shaping both my personal and professional development.
What did you do for graduation/post graduation?
I pursued a Bachelor’s degree in Computer Science from BITS-Pilani for my undergraduate studies and went on to specialize in Medical Robotics & AI during my postgraduate years at Johns Hopkins University.
What were some of the influences that led you to such an offbeat, unconventional and cool career?
I finally ended up in a career as a scientist in a camera based consumer robotics company. This journey started with me choosing a topic of study and hence my career, which has been one of the most rigorous exercises I took at that age. It was a very conscious decision on my part to do what I chose to do. Even though I did not have a formal framework to think about, in retrospect I was loosely following the holy trifecta of career decision making –
To be acutely aware of one’s strengths, the areas where one has an “unfair advantage” over the vast majority of others, the financial prospects of said areas, and the market demand associated with them in a few years, as excellently as one can predict.
After the entrance exam results were out, my parents made me talk to a variety of seasoned & young professionals, students and teachers who gave me an accurate picture of the industry expectations based on their career paths so that I could make a decision for myself. Those conversations were, more than anything, useful for me to extract unexplainable biases in people – their worldview was not the true state of my world. So, it was obvious to me that however good their intentions were, it was up to me to decide my life. Eventually I, like many others, discovered that a generalist computer science degree offered the best fit for the above three criteria I had in my mind.
My sister, apart from my parents, had a significant impact on me. She is only a year younger than me and therefore could empathise with the difficulty of my choices and at the same time observe the process for herself. Her explorative and follow-up questions helped me resolve wide inconsistencies in my thoughts leading eventually to more confident decision making. (Sailing in the same boat, she went on to become a successful Mechanical Engineer in Renewable Energy, which statistically is a much less likely choice for women, especially in Indian society.)
I guess, what I am trying to convey is that we must trust the process and ourselves, even if it is something our close family or friends may not see in the same way. Wrong choices do happen but if they do, there is no blame to be thrown around. And if our “process” is well tuned to our own self, it will eventually bear fruits.
Throughout my professional journey, Dr. Jean-Yves Bouguet, Uncle Ravi, Brother-in-law Manish, and my father have been consistent sources of guidance and mentorship. Manish, for instance, taught me the art of negotiating salaries, positions and the intricacies of the system that calls itself the immigration services of America.
Uncle Ravi, on the other hand, is my informal professional coach who nudged me towards my long-term career goals, interpersonal corporate communications etc.
Tell us about your career path
Right after graduating from BITS Pilani, I embarked on a journey with a major bank as a member of their infrastructure team. Although the financial rewards were undeniably tempting, I couldn’t help but feel that the impact of my work on humanity and on the field of computer science was disappointingly minimal. I had fallen for the gigantic machinery of on-campus placements where every opportunity is evaluated on a dollar amount, regardless of the quality or fitness of work. To me, productivity is essential, but even more crucial is the ability for that work to be a source of personal pride and inspiration. Recognizing the long-term consequences of my spending substantial time in this role, I made the bold decision to depart promptly. I had not even interviewed with anyone else by then and therefore it was a bold decision, for it was my first job ever, I was young and had no monetary cushion to fall back on.
It is my firm belief that in any career path, it’s imperative to acknowledge one’s missteps, face them head-on, and take immediate corrective action. Sacrificing short-term monetary gains for the promise of a far greater, more meaningful reward has been a guiding principle in my journey.
This proved to be a good thing for me as it provided me with the opportunity to orchestrate my first pivot — I found a rather fascinating job of a computer vision engineer for a imaging sensor startup, Tonbo Imaging (Bangalore), in the defense sector. Here I was exposed to the most advanced hardware technologies that any civilian, let alone a 21-year-old, could imagine. This was 2013, and I was mesmerized by the process of manufacturing an integrated “system” of advanced camera hardware with a tightly coupled machine learning edge-compute algorithm.
In their own words, Tonbo Imaging is a technology company specializing in the defense and space sectors. They are involved in the design, manufacturing, and deployment of various technologies, including imaging and night-vision electronics, real-time video analysis systems, 3D imaging technologies, multi-sensor imaging systems, and electro-optics systems. These technologies find applications within the defense and homeland security departments.
During my time at Tonbo Imaging, I had the privilege of being directly involved in projects at the forefront of armed forces’ operations, where our work focused on enhancing the spatial awareness of soldiers. For instance, we enabled soldiers to monitor suspicious activities along the borders of their respective countries. Additionally, we developed 3D infrared goggles designed for tank drivers, significantly expanding their field of vision to nearly 360 degrees (2π steradians) within the tank’s cockpit.
I spent several fulfilling years in this role until I reached a point where further growth required advancing my education in the field.
As I mentioned earlier, I realized the problem, and quickly acted to fix it by building a portfolio of experiences within the scope of my work to catapult me to a university where academic excellence was geared towards the larger benefit of society. My coworkers and bosses thankfully became my cheerleaders and wrote glowing recommendation letters in my favour.
I decided to pursue my masters degree at John Hopkins. One of the primary reasons for selecting Johns Hopkins, as briefly mentioned earlier, was my fundamental commitment to making a positive impact on society. The idea of enhancing a soldier’s awareness through advanced algorithms presents a complex and nuanced ethical dilemma. It involves profound considerations about the role of technology in modern warfare and its impact on human relationships. To reconcile this inner conflict, I sought to channel my efforts towards medicine, a field that I find more ethically straightforward and personally fulfilling.
Johns Hopkins University naturally emerged as the perfect choice for this endeavor. Its prestigious reputation as a world-class institution offers an exceptional environment for fostering innovation in the medical field. This is further complemented by a faculty of global renown, making it an ideal platform for me to contribute meaningfully to the advancement of medicine and its ethical imperatives.
The next pivot, therefore, was to go study Medical Robotics at Johns Hopkins University where I could collaborate with Dr. Taylor who is generally regarded as the father of medical robotics. Of course, I did not know much about Medicine or Robotics for that matter, but the three criteria I subconsciously followed 5 years earlier had come to my rescue again. I was quickly able to carve a niche for myself, thereby combining my observations from my earlier professional experiences, and my so-called ability to keep an eye on the goal of a, hopefully, successful career. I was one of the most hard-working students with many research collaborations, teaching work, publications and in some instances, patents.
I engaged in projects with distinguished professionals during this period.
Firstly, I collaborated with Dr. Austin Reiter on a Multi-instance image segmentation project, focusing on user-defined categories to address queries such as identifying a suspect in a criminal database based on characteristics like balding, Caucasian descent, age, and a forehead scar—without explicit tagging. This versatile approach could also be adapted for medical purposes, like detecting intestinal carcinoma in MRI scans of pre-teens.
Additionally, I worked alongside Dr. Iulian Iordachita, concentrating on impedance control for their lab’s robot, aimed at reducing hand tremors in surgeons. This was particularly critical for vitreoretinal surgeons who employ thin diameter instruments to delicately manipulate eye tissue while applying barely perceptible forces that could potentially harm the retina. The system effectively regulated user-applied forces on the tissue to minimize the risk of retinal hemorrhage or tears during tasks like membrane peeling, a common procedure in vitreoretinal surgery.
How did you transition to the field of Augmented Reality from Surgical Robotics?
I also had the privilege of collaborating with Dr. Marin Kobilarov and Prof. Alan Yuille. Nevertheless, my brief but impactful experience with Dr. Nassir Navab was pivotal in shaping my career. His pioneering work in Augmented Reality, combined with my background in virtual reality from Tonbo Imaging, ultimately inspired me to transition my career focus towards Augmented Reality—a decision I am committed to pursuing.
How did you get your first break?
Indeed, my transition from IT to Computer Vision was quite serendipitous. After leaving Citibank, I actively sought new opportunities. It was through a close friend from Pilani that I got introduced to Tonbo Imaging, where my friend had recently joined. During my undergraduate studies, I had covered relevant coursework in Machine Learning, Image Processing, and even authored a paper on image segmentation. While I lacked prior experience in infrared imaging, there were some noteworthy connections between Tonbo and me, including the fact that Tonbo’s CEO, Arvind, had also attended BITS-Pilani years ago. Additionally, my friend’s referral likely played a positive role in fostering our collaboration. Ultimately, my passion for computer vision and robotics was ignited by the stimulating work environment and challenging projects at Tonbo.
My first “break” after master’s, if you will, was earned through a strong referral by one of my professors at grad school. I was able to work within Google’s joint venture with Johnsons and Johnsons to build the world’s first AI medical surgeon, which would go on to mark a significant shift in medical robotics, democratizing surgery for the common man.
Verb Surgical, in their own words, is founded in strong partnership with Verily and Johnson & Johnson (Google and Johnson & Johnson Team for Robotic Surgery) to advance the surgical tools and capabilities available to surgeons and OR professionals today. Verb Surgical™ is working with physicians around the world to create the future of surgery. A new future, a future unimagined even a few years ago, which will involve machine learning, robotic surgery, instrumentation, advanced visualization, and data analytics. A future of surgery that aims to achieve improved patient outcomes, lower costs, and greater hospital efficiency.
Here I spent a fruitful summer extending the visualizations available to a surgeon.
What were some of the challenges you faced? How did you address them?
In mid-2020, amid the peak of the COVID-19 pandemic, I had an opportunity to critically evaluate government responses to the crisis. By that time, I had been residing in San Francisco for several years. Observing the reactions of both the government and long-standing residents, I encountered a challenge where I felt a sense of disconnect from that society. During times of crisis, I firmly believe in the power of a community coming together to share well-considered recommendations. Unfortunately, I witnessed the opposite approach.
Interestingly, the demand for my skills in the job market soared during this period, but my aspirations were no longer aligned with the prevailing values in my local community. This misalignment created an unsustainable situation for me. It was at this point that I comprehended why Silicon Valley is often referred to as the “golden handcuffs.” While one may achieve financial success, it can come at the cost of feeling confined within a self-imposed prison.
In response, I made the decision to free myself from these constraints and relocated to Switzerland, where the values, societal norms, and market dynamics more closely resonate with my own values, aspirations, and objectives. I would be lying if I said I was the only factor in this decision. After all, I had to take into account my partner’s career and wishes. We, together, made the move to Switzerland and continue to live here, quite satisfied with the decision.
I faced a corporate challenge when we hired a senior engineer who had been promised a leadership role by higher-ups without my awareness. Despite my relatively new role, I engaged with upper management and orchestrated a solution. We designated this individual as a leader for another team in need of management while allowing him to remain as an independent contributor to our team during their spare time. This arrangement proved highly successful, resulting in a productive two-year period for all involved.
Where do you work now? What problems do you solve?
I currently serve as a Principal Scientist and Engineer at Magic Leap, the company which is often credited with making Augmented Reality accessible for early adopters and redefining the way people think about AR. It, like most of my past career, offers me a front seat view to making a truly advanced vision enabled AI hardware system.
Magic Leap technologies encompass both hardware, in the form of the wearable Augmented Reality headset, and sophisticated software components spanning SLAM, Online Calibration, Hand Gestures, Controller, Graphics Rendering, Meshing, and more. These software capabilities are versatile, with relevance across various specialized fields, including healthcare, aerospace, and other application domains. This is the backbone of all perception services that a user has grown accustomed to being available.
Although synonymous for the purpose of this interview, I prefer to use the term Machine Learning more than Artificial Intelligence. The features above represent cutting-edge technology, with ML as the fundamental underpinning, a core element from the inception of Magic Leap over a decade ago. It’s worth noting that without some form of AI/ML, the successful implementation of any vision-based wearable device would be significantly challenging.
Furthermore, Magic Leap collaborates with and empowers third-party developers to leverage these foundational technologies, enabling the creation of industry-specific applications tailored to unique needs and requirements.
What skills are required for your role? How did you acquire the skills?
Interestingly, the academic background coupled with practical skills required to do such work intersects highly with medical robotics. These include a strong understanding of Simultaneous Localization and Mapping, sensor characterization incl for cameras & IMUs, nonlinear optimization, Differential Algebra, Lie Algebra, advanced C++ and on-chip memory layouts.
What’s a typical day like?
A typical day at work for somebody in this position could be reading through literature, communicating with cross disciplinary teams and spending considerable time thinking about, designing, and implementing algorithms on PC and on edge devices.
The ultimate test of the usefulness of anyone’s work in such an environment is to be able to make your feature as part of the final product as efficiently as possible, and ensure that it is being used by most target customers. For example, my primary concern is to estimate the deformation of the wearable AR device we make. Therefore, my target customers are all users who have dropped their headsets such that it has undergone severe deformation and therefore mis-calibrated themselves.
I oversee the software aspect of our project, focusing on deformation estimation rather than deformation minimization. My work involves close collaboration with our hardware team, specifically optomechanical engineers who assess the headset’s durability against various mechanical stresses it may encounter during its lifespan. Together, we ensure a seamless and optimal user experience throughout the entire supported lifecycle of the Magic Leap headset.
It is a highly demanding job requiring me to constantly read, be able to make approximations that can make problem solving tractable but at the same time unnoticeable to end-users. It requires cooperation by many scientists, developers, data collectors, and executives to be able to deliver high quality consistently. It is tough love and just like it, addictive.
How does your work benefit society?
My broader work is to find a way to augment the real world within a person’s eyeballs with useful and life saving information. My work gets used by surgeons in operation theatres to perform complicated procedures with minimum risk to the patient by overlaying the CT scan on the patient, also by architects to visualize the design of their building on the very empty land, much before their designs bring those lands to life with brick and mortar, by workers on a factory floor receiving instructions from their team thousands of miles away right on the instrument they have in front of them. It is meaningfully making lives easier and I am extremely proud of the small part that I play in enabling this to happen. I also contribute to the common knowledge base of humanity by pushing the boundary of what is possible with “some maths on some sensors”.
Tell us an example of a specific memorable work you did that is very close to you!
The first job experience is always memorable as it’s a learning journey into corporate dynamics. Beyond the foundational aspects, what excited me the most was the opportunity to contribute to the development of a Virtual Reality Headset for military use inside battle tanks. This was in early 2014, when the Oculus Rift (or the DK-2) had just been released. We were at the cutting edge of vision systems during that period.
I was the only person on this project and therefore this experience demonstrated to me the extent of what I could achieve even without formal training in fields like robotics, optics, or computer vision. Although, since then I have designed and implemented exponentially more complicated products, this ones holds a special place in my heart.
Your advice to students based on your experience?
My advice to students would be to spend their early years building skills in a certain niche so that they are in the top 1% of that particular topic in the world. Of course that often never happens but that should be the north-star. To answer what that niche should be, I would highly recommend the three-pronged strategy: to find the best fit and the largest intersection within the things they have an unfair advantage in, near-future market demand, and their aspirations. And develop a hobby of reading as much as you can, with a good spread of topics and interests – it always pays off to be well read. But most of all, have fun while doing it and be adequately confident in your hard-earned expertise.
The authors also went into it in great detail in their book
Garg R et al, “Classroom to Corporate”, “The AIM Model”, Clever Fox Publishing, 2023, pp. 23-40, where I contribute to the chapter with an example from my life. I highly recommend any serious aspirant to read the book in its entirety.
Future Plans?
In the future, I envision further pioneering advancements at the intersection of augmented reality and artificial intelligence. My journey has been defined by a commitment to long-term goals over short-term gains. Continuing with this ethos, I aspire to lead transformational projects that revolutionize how we interact with information and reality itself, pushing the boundaries of human-machine integration. Embracing challenges, nurturing innovation, and fostering collaboration will be the guiding principles as I strive to make technology more accessible and beneficial for society.