Almost six years ago when I started as a data engineer, I didn't know much about it and it was totally accidental. I applied for a BI analyst and I was offered a job as a data engineer followed by the shortest interview of my life during which the main question was can I do ad hoc reporting. I can. As a teacher that's all I did apart from teaching.
Anyway when I started, I wanted someone to hand me a schedule and a list of skills that I needed to learn to be a proficient data engineer, but I couldn't find one and people I was working with were not a lot of help. Ego and sexism and all.
After six years, I am in a place to make a rudimentary list of what basic skills one needs to be a good data engineer. This list is not exhaustive and will change from time to time, but it is a start.
The skills that I had when I started in data engineering:
1. Databases: I had studied databases in the university and I knew how they worked theoretically and how to create a data models. Databases include both SQL and NoSQL databases. I also learnt a little about graph databases, but never used them for work.
2. Data Warehousing and Business Intelligence: Again I had studied this as a subject in the university and was familiar with the development models (bottom up and top down). Was aware of the concepts of slowly changing dimensions, schemas (star, snowflake, and constellation), data marts, etc.
3. SQL : I knew this in theory, but was not very confident in using it. I would have an internal panic attack while using group by clause, but got over it by practicing and constantly learning.
4. Python: Learnt it from Karachi AI. Was a wonderful course, wasn't very good at it also practically, but knew the concepts and could look up code and modify it. Again got better by practicing and constant learning (honestly still learning and improving it)
5. Data fundamentals : I have loved data since my childhood, I love collecting data, sorting it and using it. I made my own datasets for courses in university and generally enjoyed working with it. So, Structured, not structured, batch, streaming, whatever kind of data, I like it and loved looking at it.
Some Non-Technical skills I have:
1. I can teach myself anything that I put my mind to. It is useful for learning new skills and improving skills that you already have.
2. I am capable of looking at problems and breaking them down in smaller, simpler problems. The credit goes to my mathematics teacher from grade 6.
3. I don't get bored by doing repetitive tasks. I am just this way. I find the repetitive tasks soothing and I usually use this time to think about other problems that I could be solving at that time.
4. I don't feel inhibition while asking for help. I ask questions, I admit when I don't understand anything and I can argue if I don't agree with something because I don't understand it or I don't see the point. This lack of inhibition has its own downside, people do presume that I am problematic and argumentative and that I disagree with them for the sake of disagreeing, but well, I can't help using my own brain.
Some Skills that I learnt on the job:
1. Tools: I learnt how to use different IDEs for accessing databases and data warehouses, So far, I have used Dbeaver, MS SQL Server, Vertica, MYSQL Workbench, DataGrip, Oracle etc.
2. SQL: SQL comes in different varieties, the SQL that I have worked most with Transact-SQL which is used by MS SQL and is the language used for Synapse data warehouse and MS SQL Server. The other frequently used is MySQL. I also used, PL/SQL, Vertica, Presto etc. The basics of all the varieties of the SQL is similar, I find that it is mostly, the date/time functions and String functions that differ, but again there is no mastering the entire language because technology change and it is continuous learning and upskilling.
3. Databricks: that is something that I learnt from scratch and loved it. It is a one tool for all of your data lifecycle needs, from data warehousing to data science loads and BI. I learnt the data governance, PySpark and other nitigrities of the techonology. There is loads to learn still.
4. MS Azure: I was aware of cloud computing, distributed computing, but it was my first time using it as infrastructe, software, and platform. I learnt to use Azure Data Factory, Azure Synapse, Datalake, Logic Apps, Machine Learning. A lot of tools that have been retired or merged with Microsoft Fabric. Again the technology evolves and the tools will retire and merge, and one has to keep abreast with it.
So this is the basic of data engineering skills that I have learnt, build upon and use in my every day job. I'll write more in detail about all the skills and some other things in the coming days.
No comments:
Post a Comment