Lazy Girl's Guide to Data Engineering

Almost six years ago when I started as a data engineer, I didn't know much about it and it was totally accidental. I applied for a BI analyst and I was offered a job as a data engineer followed by the shortest interview of my life during which the main question was can I do ad hoc reporting. I can. As a teacher that's all I did apart from teaching.

Anyway when I started, I wanted someone to hand me a schedule and a list of skills that I needed to learn to be a proficient data engineer, but I couldn't find one and people I was working with were not a lot of help. Ego and sexism and all.

After six years, I am in a place to make a rudimentary list of what basic skills one needs to be a good data engineer. This list is not exhaustive and will change from time to time, but it is a start.

The skills that I had when I started in data engineering:

1. Databases: I had studied databases in the university and I knew how they worked theoretically and how to create a data models. Databases include both SQL and NoSQL databases. I also learnt a little about graph databases, but never used them for work.

2. Data Warehousing and Business Intelligence: Again I had studied this as a subject in the university and was familiar with the development models (bottom up and top down). Was aware of the concepts of slowly changing dimensions, schemas (star, snowflake, and constellation), data marts, etc.

3. SQL : I knew this in theory, but was not very confident in using it. I would have an internal panic attack while using group by clause, but got over it by practicing and constantly learning.

4. Python: Learnt it from Karachi AI. Was a wonderful course, wasn't very good at it also practically, but knew the concepts and could look up code and modify it. Again got better by practicing and constant learning (honestly still learning and improving it)

5. Data fundamentals : I have loved data since my childhood, I love collecting data, sorting it and using it. I made my own datasets for courses in university and generally enjoyed working with it. So, Structured, not structured, batch, streaming, whatever kind of data, I like it and loved looking at it.

Some Non-Technical skills I have:

1. I can teach myself anything that I put my mind to. It is useful for learning new skills and improving skills that you already have.

2. I am capable of looking at problems and breaking them down in smaller, simpler problems. The credit goes to my mathematics teacher from grade 6.

3. I don't get bored by doing repetitive tasks. I am just this way. I find the repetitive tasks soothing and I usually use this time to think about other problems that I could be solving at that time.

4. I don't feel inhibition while asking for help. I ask questions, I admit when I don't understand anything and I can argue if I don't agree with something because I don't understand it or I don't see the point. This lack of inhibition has its own downside, people do presume that I am problematic and argumentative and that I disagree with them for the sake of disagreeing, but well, I can't help using my own brain.

Some Skills that I learnt on the job:

1. Tools: I learnt how to use different IDEs for accessing databases and data warehouses, So far, I have used Dbeaver, MS SQL Server, Vertica, MYSQL Workbench, DataGrip, Oracle etc.

2. SQL: SQL comes in different varieties, the SQL that I have worked most with Transact-SQL which is used by MS SQL and is the language used for Synapse data warehouse and MS SQL Server. The other frequently used is MySQL. I also used, PL/SQL, Vertica, Presto etc. The basics of all the varieties of the SQL is similar, I find that it is mostly, the date/time functions and String functions that differ, but again there is no mastering the entire language because technology change and it is continuous learning and upskilling.

3. Databricks: that is something that I learnt from scratch and loved it. It is a one tool for all of your data lifecycle needs, from data warehousing to data science loads and BI. I learnt the data governance, PySpark and other nitigrities of the techonology. There is loads to learn still.

4. MS Azure: I was aware of cloud computing, distributed computing, but it was my first time using it as infrastructe, software, and platform. I learnt to use Azure Data Factory, Azure Synapse, Datalake, Logic Apps, Machine Learning. A lot of tools that have been retired or merged with Microsoft Fabric. Again the technology evolves and the tools will retire and merge, and one has to keep abreast with it.

So this is the basic of data engineering skills that I have learnt, build upon and use in my every day job. I'll write more in detail about all the skills and some other things in the coming days. 

Sugandh

Read a book

Just watched a bunch of videos while knitting and relaxing and then found out that people are talking to chatbots because they are lonely. 

Don't talk to a chatbot, go read a book!

We read to know we are not alone. ~ C.S. Lewis

Sugandh

Truncate vs delete from

Sometimes I make weird rookie mistakes when working with SQL and one of them is confusing syntax of various commands. Today it was truncate.

So the syntax is

Truncate table table_name;

It doesn't have a where clause and is used to remove the entire data. On the other hand when we use delete, the syntax can be

Delete from table_name;

It will delete the entire data, but we can also use a where clause.

Delete from table_name where column_name = criteria;

I was using truncate table table_name where date = cast(getdate() as date);

Wouldn't work and will definitely fail.

Sugandh

The Integration Tables

Hi, 

From the depths of a corporate office, where I write SQL queries for a living. When I was studying data warehousing and business intelligence - which was eons ago - we had staging tables and the productions tables. 

The staging tables consisted of unaltered data. Data directly from source and then you can manipulate data - transform/clean/ standardize it and then load it to the production tables that could be used by business intelligence or knowledge mining teams.

Yesterday I found out that staging tables are now called integration tables in Microsoft-verse. With a slight addition that since there could be various data sources integration tables are used to consolidate the data from all those sources. It was also done earlier, but somehow the powers that be needed to clarify this.

Hope you have a great day.

Sugandh

Setting Up A Kitchen

The most difficult thing that i needed to do when I started living alone was set up my kitchen. Living alone happened in two phases, once when I moved out for studies and once when I got married. 

Setting up kitchen for a single person is relatively easy as it is only you, you are responsible for. Back then, I could cook like three things and lived on those. I would make a huge pot of pasta and eat it for next three meals. At times, I'd have corn flakes for dinner. Afternoons were spent at university - so cooking wasn't a huge part of my life.

In terms of spices, I had the following.

Red chili powder

Black pepper

Salt

Oregano (pasta!)

Ground Coriander seeds

And that's about that.

The second phase was way more different than the previous time. I was responsible for running a grownup home, and I was anything but. I knew the basics. I needed pots and pans, stirrers and spatulas, some containers and that's about that. What I didn't realize that a functional kitchen has much more than I thought it would need. There are pots for cooking rice, meat and vegetables and there is a pot to make sweet dishes or boil milk in. There are different size pots for making tea and smaller pots for warming up food. A frying pan, deep enough for deep frying and a smaller one because the bigger one would take too much oil. a container for oil, plates - a lot of plates.

I bought a chakra belan (a rolling pin and a rolling board) and a container for the flour, but I forgot to but a dish big enough to knead dough. That I bought second time. I also realized that I needed strainers and colanders of different sizes for draining tea, rice, pasta etc. and then there were organizers, a container to store small gadgets - measuring spoons, measuring cups, funnels, a tiny grater and a bottle opener.  On top of that my kitchen doesn't have drawers, so I have to get containers for cutlery and other stuff.

When it came to cooking there are more spices needed than the ones, I was familiar with. For example, I needed whole spices and garam masala and turmeric etc., I needed ginger-garlic paste and soy sauce and chili sauce and mixed herbs, just to make food that I wanted to eat.

Then there were baskets, for fruits and for other things. Packets of pasta, rice, spices, jars for jams, bottles for sauces. Just to keep things organized. It was also necessary for me to train my husband in the art of organization in our kitchen. I don't personally like men who knows nothing about their kitchen and can't find a spoon when needed. 

Sometimes, when I was trying a certain recipe, I would realize that I am missing a certain spice, and I would decide to move ahead without it. The food was edible, and not perfect at all. I would add that spice to the shopping list for next time. I also had to collect recipes that we both enjoyed eating and were easy to cook. 

Through all of this, all I wanted was a list of things for people who were starting new home from scratch. What are the must haves. Here I am making a list from my own experience.

Things to have in a starter kitchen:

1. Depending on people pots of different sizes: for rice/pasta/noodles, for meat and vegetables - things cooked with spices, for sweets and milk/tea. 

2. Strainers and Colanders

3. Baskets for fruits (I have in different sizes)

4. a gridle (tawa)

5. a bowl/platter for kneading dough (I forgot this one)

6. mixing bowls

7. frying pans

8. containers for spices/ lentils/rice/flour etc.

9. Containers for storing half used packets of stuff.

10. spatulas and turners.

11. Wooden spoons for cooking and mixing

12. beaters and whisks

13. plates and glasses

14. serving bowls

15. cutlery

16. serving spoons

17. Rolling pin and rolling board/mat

18. Cutting board

19. Knives different sizes and knife sharpener

20. Blender and a grinder

21. pestle and mortar

22. measuring cups, measuring spoons, measuring jug

23. Water bottles,

24. Oil bottle/jug

This list is not at all exhaustive. Depending on your lifestyle and the kind of cooking you do, things can be added or subtracted. 

If you are reading this and making a new home, I hope that your home brings you happiness and joy and you enjoy setting up your kitchen and using it.

Thoughts on Being Famous and Self-Promotion

I have never wanted to be famous. I have never wanted anyone to know me. I have wanted attention from a handful of people in my entire life. I crave anonymity and solitude. But for the last few days, I have been thinking about self promotion a lot.

I have been seeing these people write articles about things that there are thousands articles on. I have been seeing people at work list the tasks that are everyday maintenance work as things they did. I have been noticing people post their pictures with famous people to assert their importance and flaunt their knowledge. There is so much self promotion around me that I have been feeling inadequate. May be it is the first time I am seeing people who know more than I ever will or they are just good at saying that they know more when in reality all they do is know as little as I do.

This begs the question: what do I want to do in this situation? Change who I fundamentally am (was) because I do crave recognition for my work and skills, or just let things be and continue doing what I have always been doing (not care for anything).

Life has so much been about learning new things every year, but the most difficult are the things that one learns about oneself. There you are secure in thinking that you are basically a good person and not need change, but then you come across a scenario that you never thought about and you question the philosophy that you have painstakingly curated throughout the life. Your fundamental beliefs about yourself change and you realize that you are not who you thought you are.

So this scenario around me - where everyone is blowing their own trumpet - where does it leave me? Do I still need to be anonymous as I always wanted to be, or do I change and ask for recognition in the gaudiest of the ways possible (for me) to get the recognition that I crave.

Sugandh

A Room of One's Own

When Ahsan and I were looking for home, even before we were formally married, i insisted to have a home with an extra room, so that i could have a room of my own, where i could be as wild as i want to be. That wouldn't be accessed and judged by anyone coming from outside. A room that will be the implementation of my wildest dreams and the finest of the luxuries that i so desired. Room that is reflection of my thoughts, ideas, creativity and would house my women, my dolls and statues. 

I have that room now. It is not yet as wild as i want it to be, but I'm slowly getting there. It is my refuge and manifestation of me. I love my entire house but i love my room the most.