Real world applications of Computer Vision will have a truly global impact on our society through smart automation driven by precision engineering !

Subhayan Mukherjee, our next pathbreaker, Computer Vision Research Engineer at Blue River Technology (California), wholly owned by John Deere, works on software and hardware that is used to make tractors autonomous.

Subhayan talks to Shyam Krishnamurthy from The Interview Portal about his diverse career experiences starting from his first job at Informatica (a software product company), and shifting gears to do a PhD in Computing Sciences at the University of Alberta (Canada) where he worked on several applications of machine learning in healthcare, media and remote sensing.

For students, pick up diverse experiences along your way so you can fine tune your career path based on what you like doing genuinely !

Subhayan, tell us what were your early years like?

I grew up a single child in a middle-class household in South Kolkata. Ours was a joint family, and my folks still live in our ancestral house. We have always held education in high regard, although I would not go so far as to say we are hyper-careerists. Our relatives are a mix of professionals (in both tech and non-tech fields), and businesspeople, carrying on family businesses or trying to further those that they built from scratch. My father had an electronics background and was a senior technical officer at what is now a PSU under the DAE. My mother has been a homemaker.

I was fortunate to have attended one of the best schools in the city. Although it was not apparent to me back then, in hindsight, it seems my schooling and peers had significant positive influences on me. I was not career minded till I got into the industry after graduation. However, looking back, I realize that the academic rigor of my school played a major role in ensuring that even “not-so-studious” folks or those that did not make it into the top percentiles of the class every year still had a solid foundation for them to try and build a career, should they choose to, later in life. My extra-curricular interests included painting and computer programming. This was also partly due to my school that made these activities compulsory (among others) from second to eighth grade. Most of the awards I won till twelfth grade were in these two extra-curricular subjects, and I can clearly recall my teachers nudging me and signing me up for all school- and district-level competitions on painting. It is not hard to view what I do now for a living, as the result of pursuing a combination of these interests.

What did you do for graduation/post graduation?

I did my Bachelor’s in Information Technology (BTech) from a leading autonomous private trust-funded institute in Kolkata affiliated to the West Bengal government’s technological university. My post-graduate degree (MTech) in the same field (IT) was from NIT Surathkal, a leading public institute near Mangalore, Karnataka.

I then did my PhD from University of Alberta (Edmonton, Canada) in Computer Science.

Can you talk about significant influences that led you on such an offbeat,unconventional and uncommon career path?

My BTech final semester project was on satellite image processing, and the institute professor who mentored us was very particular on not making the project simply an implementation of one particular algorithm, but rather, experimenting with multiple ones. He was also very particular about us presenting a detailed comparison of the results of different algorithms to solve the problem at hand, what the pros and cons of each were, etc. He always emphasized on research-oriented projects.

I think one turning point in my career was the 2008 economic recession. I was not immediately affected by it since our engineering batch was the last one that had campus placements in their third year. We thus got our first job relatively easily, before the full effect of the recession was felt in the job market. However, none of us were blind to what happened to our immediate juniors in terms of job placements, as the economy went further into recession. Our joining dates were also delayed by months, and all of a sudden, the few of our batchmates who had prepared diligently for post-graduate studies looked like the ones that had made the smartest decision. I had two placement offers from the leading software consultancy firms in India, and went ahead with the one with the earlier joining date, Infosys, even though that was not in my city. Although the five-month training was memorable, the lack of projects from US clients who were mired in recession meant limited learning opportunities on projects after the training period. Also, I was not particularly excited about the nature of work in most projects. The type of work that was generally off-shored from the US to software consulting firms, back then, was mostly about repeating the same type of maintenance on software that was already built primarily in the US. Thus, there wasn’t a lot of diversity and scope for influencing core software design decisions. Lastly, most projects were about maintaining or supporting software custom-built for the specific client, and not software that is widely used in a particular industry.

All these prompted me to start preparing for further studies, in order to specialize in some area that interested me to be able to further my career in that area. This, I reasoned, would give me the deeper insights needed to design software that is able to address a wide variety of challenges faced in the real world and serving an entire industry, rather than specific use-cases for a particular client. In common terms, we refer to companies that design such types of software as “product” companies, as opposed to “service-based” companies that maintain or build on software built to handle specific client requirements.

How did you plan the steps to get into the career you wanted? Or how did you make a transition to a new career? Tell us about your career path

Once I decided to pursue postgraduate studies, I started preparing for the post-graduation entrance exam and practice tests, and eventually, based on the exam score, got a call for a written test and interview from NIT Surathkal, where I got selected. It was a research-based masters, with roughly one semester of coursework and research for the remaining three semesters. I chose a topic that involved trying to simulate a technique used in portrait photography that isolates the subject of a photo by making the background out of focus. This involved estimating the distance of the subject and the background, and using a pair of images of the scene shot from two slightly different angles. There were very few public datasets back then that had such types of images, but my project guide made it a point to acquire cameras that could capture such pairs of images ( called “stereo” cameras ) using the institute’s research funds. I was able to publish a journal, present a conference and workshop paper based on this project. 

These helped make my PhD application stronger. The professor who was my masters project guide played an important role in preparing me for PhD admissions. Since he got his PhD from a reputed university abroad, he had first-hand experience of how PhD applications are evaluated, how research papers are reviewed, how to decide the reputation of a conference or journal, and how to choose universities and professors for PhD applications. Thus, his guidance helped me.

A few months into my masters programme at NITK, my project guide asked me about my future plans. I realized during our discussion that to be able to get accepted to a reputable PhD programme, I would have to build a profile that shows some evidence of research potential. The most practicable way to do that was to have a thesis in a research area that is gaining prominence, and to have a masters project with results published in at least a conference that is recognized nationally or internationally, besides a journal, from a reputed publisher. However, to get these ready before the start of the application cycle for admission a year later, would most likely mean not being able to start my PhD programme right after completing my masters. I figured that the path of the best compromise between risk and reward would be to gain some experience into how product companies work, in parallel to gathering credentials needed for a strong PhD application. The reason for this is that placements at NITK, and probably all such institutes in India, start shortly after the start of the final year of the degree programme. That would mean, that in the highly unlikely scenario that I get a PhD admit before that, passing up an opportunity to gain experience at a product company, and just preparing PhD applications and waiting to get into a programme, being otherwise unemployed, would likely not be good for either my career or resume.

I accepted an offer from Informatica during campus placements at NITK. Although opportunities in the particular field I was interested in and had my thesis on were quite limited back in those days, I cherish the work culture and technical experience I gained at Informatica, which was the first, and only product company I worked at, for a year before my PhD.

Eventually, I got accepted into, and earned my PhD from the department of Computing Science at a leading public research university, University of Alberta in Edmonton, Canada. During my PhD, I worked on a combination of problems that have a common theme of efficient extraction of areas of interest from large collections of images, aggregating and correlating them in order to find meaningful information that could help in decision-making. One example would be estimation and tracking of brain injury volume from MRI scans for pre-term infants. I worked with clinicians from the Hospital for Sick Children, Toronto for this project. Magnetic resonance imaging, abbreviated as MRI, scans tissues inside our bodies as separate images vertically stacked on top of each other, like a pile of paper, where each sheet represents each image, or each MRI “slice” of the organ being scanned. Thus, the MRI stack represents the MRI data obtained for the entire mass of tissue making up the organ, e.g., the brain, in three dimensions. This is useful to get a 3D representation of the brain, and to track how an injury to one part of the brain grows or shrinks over time due to lack of or in response to treatment.

In an internship with a company 3vGeomatics, Vancouver, during my PhD, I worked on predicting landslides by tracking fine ground movements from images captured by radars mounted on orbiting satellites. This project also represented a turning point in my career as this was the first time I got seriously involved with machine learning, having only taken two courses on the subject earlier during my PhD. Also, I think my PhD supervisor had quite a bit of foresight to infer the impact machine learning, and especially deep learning, would have on the field of computing, since it was still in the early stages of gaining prominence back in 2015 when I joined my PhD programme. 

It was through this internship that I gained practical hands-on experience with a type of algorithm that is widely used in machine learning nowadays, called convolutional neural networks. This experience also helped me land my first job after graduating in April 2020, which was arguably the worst time to graduate, right after the onset of the global pandemic. I was being considered for a permanent full-time position with 3vGeomatics, but they were forced into a hiring freeze in February, just over two months before my PhD defense. Luckily, shortly before my defense, a recruiter from the Edmonton office of an engineering consulting firm contacted me on LinkedIn for a permanent position requiring the type of background I had, and I got selected.

During my PhD, I also got an internship opportunity at a leading industry lab (Dolby Laboratories) working on high dynamic range imaging in Sunnyvale, California. The work involved designing efficient algorithms that would hide image artifacts created when a low dynamic range image or video is viewed on a high dynamic range screen. Although HDR TVs and mobile phone displays are becoming common nowadays, back in 2016, they were still not widely available (or affordable) in the electronics consumer market in the US or elsewhere. A HDR display is capable of reproducing a wider range of light intensities than a LDR display. This creates an experience of an image that is closer to how we see it in the real world, and is often crudely described as “darker darks and brighter brights”. I was able to get a patent and write a conference paper based on the work there. My eight-month stay and trips to San Francisco, Monterey, Carmel, Santa Cruz, etc. were among my most memorable.

How did you get your first break?

After my PhD thesis defense, I started working at the Edmonton branch of a leading, global consulting and engineering company, Tetra Tech, that has its headquarters in Pasadena, California. Although initially, I had my reservations when the recruiter contacted me over LinkedIn, I soon realized, after the interview with the CTO, that the goal here was to build software for the railroad industry in general, and not for specific clients. 

What were some of the challenges you faced? How did you address them?

So, the division inside Tetra Tech that the CTO manages operates like a product company as opposed to an engineering consulting firm. We had our software (and hardware) that allows rail corridor monitoring based on different types of sensors fitted to the exterior of a special rail car that we also design. As the car travels with the rest of the train, live feed from the sensors are analyzed to detect defects and normal wear and tear of the rail tracks, in addition to obstacles like tree branches that are at risk of falling on the rail track. 

This was also a job where I started experimenting with other types of machine learning models that I had not used previously. Thus, one of the challenges was to be able to learn about those models, and how to leverage them to implement an acceptable solution for the industry problem at hand in a relatively short amount of time. There was also the challenge of having to learn a new programming language, so as to be able to integrate this solution with the rest of the software. During this period of few weeks to months, I learned the skill of grasping new concepts that came up while I was trying to leverage unfamiliar areas of research using technical blogs like “toawardsdatascience” and “medium” instead of trying to go through a series of full-length research papers on the models and trying to understand all intricate details of the theory. Similarly, I learned a good deal about the importance of modularity, unit testing, etc. while implementing my initial solution using an unfamiliar programming language, to the extent of having it run on the railcar in three weeks.

Where do you work now? What problems do you solve?

I currently work at “Blue River Technology” which is wholly owned by John Deere. We build software and hardware that is used to make tractors work like self-driving cars, and spray chemicals on farm fields very precisely, depending on where they are needed. So, for example, if there is a crop that is very close to a weed, the software will direct the nozzle to spray the pesticide only on the weed, and not the healthy crop. This determination is based on the live camera feed from cameras installed at strategic locations on the tractor. We determine how all those cameras need to be placed, what the specifications of the individual components of the cameras need to be, what type of algorithms need to be used to analyze the video feed coming off those cameras in real-time, on the tractor.

What skills are needed for your role? How did you acquire the skills?

Other than computer programming, computer vision, machine learning, and signal processing are among the main skills needed for the software side of things. I acquired those during my courses and internships.

What’s a typical day like?

My workdays involve a mixture of programming to automate manual tasks, some experiments and implementation of new features or improving existing features of the software. There are meetings with my manager to discuss personal, career development or work-related issues. There are also meetings within my immediate team, as well as my broader team. In such meetings, people usually discuss some design decisions or just some general updates on what we are working on at an individual level, and present some of that work to others in the team. Sometimes, there are all-hands meetings where we get to know about the broader directions the company is headed towards, in terms of both technology and sales, etc. There is usually quite a bit of coordination between different teams, often across different divisions.

What is it you love about this job?

I can say without exaggerating, that my current job is one of the most rewarding experiences I have ever had in terms of both company culture and learning. I get to experience the whole chain of processes that comprise the broader task of image analysis, both in terms of hardware and software. This is something I have never had anywhere else, and has given me a practical exposure and understanding of how a lot of image processing and computer vision models that I learnt in separate courses or industry projects, fit together to solve a real-world problem.

How does your work benefit society? 

All our products improve crop yield and quality. They drastically reduce pesticide usage, as pesticide gets washed off to rivers and severely pollutes the water. Hence the name “Blue River” as that is our objective. Tractors running autonomously free up farmers’ time, which they can use to find better markets for their produce. We also make it possible to run harvest 24/7, which is both inhumane and prohibitively costly if done by seasonal farm workers, a major challenge in the US. The platform provided by John Deere enables us to have a truly global impact on mechanized farming.

Tell us an example of a specific memorable work you did that is very close to you!

In the recent past, there was a project that involved making images robust to lighting changes, such that human annotators would be able to mark features of interest on those images that the algorithms need to focus on. The algorithm uses such features or points of interest on the image to learn relationships among several images that show those points from different viewpoints. This becomes particularly challenging for images with poor lighting conditions, e.g., those shot at dawn or dusk, or those shot amidst dust, fog, rain, or cloudy conditions, all of which are possible out in the fields where a tractor runs, but not in an everyday well-lit indoor environment. One particularly difficult scenario is when there is sun shining directly into one or more cameras. The challenge here was to design an algorithm that would run fast enough such that it does not become impractical to use for a human who would review such challenging images as the tractor collects them, to provide feedback on how efficiently the tractor is actually able to run autonomously for a long time duration. Farms in the US and other parts of the world could have a very large area. Thus, for an autonomous tractor to be useful, it should be able to run for several hours on end without a human operator’s intervention for any reason whatsoever. 

Your advice to students based on your experience?

Grades and institute reputation do matter, but are not sole determinants of, or sufficient for career success.

It helps to try and find what you like doing genuinely. You will most likely face challenges and pitfalls when you pursue a career. If you like what you do, it will be easier to deal with those, rather than if you choose a career blindly, exclusively based on others’ opinions.

Learning does not stop after obtaining X, Y, Z degree from X, Y, Z institute in your home country or abroad. If your career advances, you learn every day at work.

Peers and mentors matter, so choose them carefully.

Future Plans?

It seems we will continue to have very good career development opportunities in autonomy, given the developments I see in my current role and in the industry and academia in general. So, for the foreseeable future, I see myself growing in this field, seeking out opportunities to pick up new skills, and make the best out of, and build on those that I already have.