Technologies such as Data Science, Machine Learning and Predictive Analytics are disrupting the way businesses and governments will be run in the future.
But these technologies are only as good as the underlying data that they process. Our next pathbreaker Shilpa Arora, Data Scientist at Atlan, ensures that businesses have the most relevant, accurate and timely data to build models that help understand customers, demographics and segments that can be precisely targeted for sales, marketing or policy making.
Shilpa talks to Shyam Krishnamurthy from The Interview Portal about how her love for mathematics and numbers helped her choose the kind of career she wanted and what she loves about her work with Data.
Tell us about your background
I was born and brought up in Ghaziabad, UP. I was pretty clueless about what I was going to do in my life post school but I sort of enjoyed maths so I took commerce and maths. And thanks to Delhi Times newspaper, I got to know about the “cool” life of DU kids and badly wanted to be one. So going to Delhi University (DU) was decided, and I worked really hard to get good scores.
And luckily for me (and only me) the 2008 recession came when I was in 12th standard which spurred my interest in economics. So I pursued Economics (honours) from Indraprastha College for Women, DU, and then did a masters in economics from Madras School of Economics.
How did you end up in such an offbeat, unconventional and interesting career?
My career is a combination of multiple influences and events, and it is still evolving. This process will be different for everyone but there are a few decisions that really helped me in choosing jobs that just don’t bring money but also teach me and make me happy everyday-
- What I like and what I am good at are two separate things- While studying, I liked almost all my courses but I was good at only 50% of the subjects like statistics, econometrics, game theory, macro modelling, etc. which were mathematical in nature. So while working, I played to my strengths and chose job profiles that had numbers (data) as their core and learnt other things on the side. This way, all my mentors were willing to invest their time in teaching me things I was terrible at because they believed they could rely on me completely on other things and of course, I am lucky to have had and still have such great people around me.
- Trial and error- I did a lot of internships and kept picking new things while on the job or beyond to know what really interests me, what I am good or bad at. For example, doing an internship in development economics with IGC, a research centre of Oxford and the London School of Economics (LSE), before taking my first job helped me understand what I wanted my first job to be. I realised I enjoyed sitting and analysing data more than travelling and taking interviews of IPS officers or designing presentations and questionnaires with professors. I basically preferred the fast paced work life of a startup over academia, which was my dream in college.
Tell us about your career path
As I mentioned earlier, I did a bunch of internships- one in financial market, one in development sector, and one in a research centre of an external university to understand work in these fields. They helped me understand what I didn’t want, not what I wanted but that was enough to start. And hence, I didn’t sit for my campus placements and decided to do a decently paying internship for 2 months after college.
While doing that internship, I got an offer from a company, Indicus Analytics, which seemed to do the kind of work I thought I would enjoy- it was a data company which worked on different types of data, I was joining a product team which meant they would care about code quality, quality of statistical models, and won’t be restricted to one or few use cases. I learnt quite a lot there but the growth seemed slow, it was little away from modern scalable concepts like ETL, ML, etc. as well as application in the real world. So I left without any opportunity in hand and then started panicking and writing to people and places I associated with.
I then joined a startup, SocialCops (now Atlan), which was fast paced, used latest data and tech concepts and, had real-world applications. And it’s been 4 years at Atlan now and everyday is a new learning.
What were the challenges? how did u address them?
- Challenge 1: While starting my career, I had no background in coding and data science requires coding in at least 1 language, so learning a programming language and evolving coding skills with new data, new methods and tech infrastructure has always been a challenge. To address this issue, I force myself into taking on some work that requires learning that language or related skill and putting myself into pressure. I seek help from web- mostly stack overflow, sometimes do an online course, if required, and also, ask around my colleagues and keep practicing.
- Challenge 2: I have always been a quiet, introverted, and under confident, so communicating effectively has always been a challenge for me. But a big part of being a data scientist is being a great communicator as you constantly have to communicate with business teams, engineering teams and multiple external clients. And this was a big hindrance in my growth as I was not even able to communicate effectively with my team. But then, I decided to break down what communication really means, spoke to my mentor and read a few books as well.
And I learned that communication is about thinking and preparing as much as possible before speaking, it’s about maintaining clarity which becomes way easier if you quantify- like don’t say “I will pass you the file in sometime”, say exact time like 5 pm or at least give a time window. It is about reading and writing on a daily basis, which helps you articulate better, it’s about being sure of your facts and opinions as you genuinely will be confident then. So, I started picking one such attribute every month and focused on constantly putting them to action, and I still constantly work on it.
Where do you work now?
I work as Principal Data Scientist at Atlan (Atlan | Home for Data Teams). Recently I have been working on building insightful data products which provide access to curated, updated, granular data sets and help users enrich their understanding of their customers or a region. As an example, one of our data API provides affluence and income slab estimates aggregated at building level for all cities and villages in India which can help businesses to help market their products strategically or banks can use them to make better credit scoring models, etc.
I work on following things broadly-
- Looking out for new open datasets, data partnerships, better statistical and ML methods and models, new tools and services to support our data needs, etc.
- Build scalable methodologies that can be automated as a pipeline to generate most updated data at fixed frequency.
- Hire team members and mentor data scientists in the team.
- Work very closely with clients/users so they are able to use the data APIs in their use cases before scaling up.
How does your work benefit society?
Data allows decisions to become more transparent and evidence-based, and combining it with statistics, mathematics, computer science and powerful machines allows such decisions to become more efficient and reduce wastage of useful resources and time.
And I have been lucky to witness this first hand. For example, we built an algorithm to find the right location to open LPG centres under Ujjwala scheme all across India in a way that businesses are profitable and villagers get access to LPG centre within a distance of 5km. Without data, this would have been based on human intuition, which is very subjective and would have taken more than a year to complete.
Tell us an example of a specific work you did that is very close to you!
In the last 2 years, we have built a data product called Atlan grid and experience of building that product was like building a startup from scratch. During this period, I got to design and build the product, learn about so many different types of data, supporting ecosystems like cloud, APIs, ML algorithms and so much more, build and manage a fabulous team, visit the users and make sure every user is able to drive maximum value out of the product, and many more things around building a product and a team. I have never had such a steep learning curve in life so it is very close to my heart and I hope to only get better from here.
Your advice to students based on your experience?
1. Try learning by doing- You will learn way faster as you meet with situation not written in any book. Do internships, work with your professors or seniors, feel free to write to people that inspire you (mostly people respond).
2. Google more- It seems weird but people don’t google and find answers and opportunities. If you want to learn something, just google it, mostly likely you will find a course or book or person that can teach you.
3. Try to minimise the gap between your action and aspirations- Do this everyday, self correct yourself in the right direction. Make sure you say things, do things, and questions things that take you closer to your goal.
In the near future, I am excited to build intelligent data products that will allow data teams around the world do their best work. Building intelligent products like data cataloging systems will allow data teams find their data as easily as we find a book on amazon, it will allow sharing data as easily as we share text over WhatsApp. This will take all my learning of past 5 years so I am pretty excited to see how I can use all the knowledge I have gained to power products that are going to be used by data teams around the world.
I also plan to explore more futuristic application of data in the field of longevity and neuroscience.