+972 52-548-6969
Expanding your horizons
One of the questions that I routinely get asked is “how do you learn”. And the answer that I keep giving is that I had accidently started learning things from the basic building blocks. I still count a C/C++ course that I took over a decade ago as one of the chief reasons why I have a good grounding in how computers actually operate. During that course, we had to do anything from building parts of the C standard library on our own to construct much of the foundation of C++ features in plain C.
That gave me enough understanding of how things are actually implemented to be able to grasp how things are behaving elsewhere. Digging deep into the implementation is almost never a wasted effort. And if you can’t peel away the layer of abstractions, you can’t really say that you know what you are doing.
For example, I count myself ignorant in all manners about WCF, but I have full confidence that I can build a system using it. Not because I understand WCF itself, but because I understand the arena in which it plays. I don’t need to really understand how a certain technology works, if I already know what are the rules it has to play with.
Picking on WCF again, if you don’t know firewalls and routers, you can’t really build a WCF system, regardless of how good your memory is about the myriad ways of configuring WCF to do you will. If you can’t use WireShark to figure out why the system is slow to respond to requests, it doesn’t matter if you can compose an WCF envelope message literally on the back of a real world envelope. If you don’t grok the Fallacies of Distributes Computing, you shouldn’t be trying to build a real system where WCF is used, regardless of whatever certificate you have from Microsoft.
The interesting bit is that for most of what we do, the rules are fairly consistent. We all have to play in Turing’s sand box, after all.
What this means is that learning the details of IP and TCP will be worth it over and over again. Understanding things like memory fetch latencies would be relevant in 5 years and in ten. Knowing what actually goes on in the system, even if it at a somewhat abstracted level is important. That is what make you the master of the system, instead of its slave.
Some of the things that I especially value, and that is of the top of my head and isn’t a closed list are:
- TCP / UDP – how do they actually work.
- HTTP – and implications (for example, state management).
- The Fallacies of Distributed Computing.
- Disk based storage – efficiently working with it, how file system works.
- Memory management in OS and your environment.
Obviously, this is a very short list, and again, it isn’t comprehensive. It is just meant to give you some indications for things that I have found to be useful over and over and over again.
That kind of knowledge isn’t something that is replaced often, and it will help you understand how anyone else has to interact with the same constraints. In fact, it often allows you to accurately guess how they solve a certain problem, because you are aware of the same alternatives that the other side had to solve.
In short, if you seek to be a better developer, dig deep and learn the real basic building blocks for our profession.
In my next post, I’ll discuss strategies for doing that.
Comments
Yeah it is shocking that even in 2011 the inner workings of all of those systems still matter because their abstractions are leaky. Fortunately, brutal CPU power is often able to shield the programmer from some of that stuff. But not always.
Another oldie but goldie: DNS (scalability/caching)
Taking into account that a new year is coming and many people are pondering about their future carreer, it is a very appropriate post. It would be great to hear about building blocks of our profession, thank you in advance.
P.S. Indeed, despite having a MCTS certification, I have a very vague ideas of what is going on behind the scenes
Agree: you have to learn the concepts, then you'll know how stuff is built.
There are indeed dozens of these basic points, for a .NET programmer I would add: -> value vs. reference CLR types duality, the why and the how -> character encoding normalization -> hash table implementation -> core Windows threading and sync object management -> win32 and COM basics -> bitmap encoding strategies
Sometimes returning to the basics is an eye-opener. For example you discover that people keep reinventing the same things over and over. Document databases, map & reduce, functional programming, etc, many of these concepts have been 'invented' 30 - 40 years ago and forgotten because of more popular and simpler technologies.
100% agree, great post. I always tell people that without a deep understanding of the basic of programming you cannot really call yourself a good programmer.
Even if .NET is a managed environment, I always suggest people to have a good understanding how memory allocation works, how the operating system manages memory, etc etc, because this is the "ground" where you can construct good knowledge.
Seriously? There is no reason that people can't use WCF to build a distributed compute system without knowing all the ins and outs of these topics. Would it help build a better system? It probably would, but it also might leave a programmer so far down in the weeds that they forget the purpose of the system.
Programmers need to remember why they are building systems in the first place, which is more than likely to make someone money. Try going to a CIO and tell her you want to money to train your developers on the innerworkings of TCP/IP, disk storage and memory management. She would want to know what return she would get on her investment. I think you would be better off saying you want to train your developers on the business they support so they can build systems better, faster and cheaper in order to deliver value to the shareholders.
These things make sense for someone like you Ayende, but you are a consultant, not a programmer. You are the guy that comes in and fixes problems when something is broken, or takes things to the next level. These down in the weeds things are nice to have, but lets face it, you are smart guy with a lot of experience and that is what sets you apart.
P.S. You might want to add threading to that list as your site is taking up a full core whether I use IE or Chrome.
I have found the best way to keep growing is to never stop asking questions, and to never assume you understand much. Humility is a powerful educator.
I've also learned that listening to people that are a lower skill level than I am is useful. I learn a lot just by listening to the questions of students and amateurs, because they have grown up a few years later and are enthralled with facets of the programming languages that were not as big when I was in school. For example, when I went to school, memory management was a very big topic, as was RDBMS, but now if you sit and listen in on a college course there is a lot of talk about cloud computing and disconnected data.
Learning is what a programmer is good at. I don't judge a programmer by their ability to write code exclusively, I judge them by the ability to be handed a project, and research it, and the humility to go ask smarter people how they solved it if needed.
My most important skill: Knowing how to use a search engine to quickly find the right answer(s).
In my case it also helps I'm a server system administrator (both Linux and Windows), so I have pretty good knowledge about low-level netwerk specifications. I always want to know what makes an application or service tick.
And that brings me to my third option to learn, and that is to read how others (like Ayende, Chris Brown or Dominick Baier) try to solve issues. That's why I like open source, you'll often read a few comment lines why they have decided to thing a certain way and choose not to implement option x like th epost were Oren explain why they had decided to alter a existing JSON parser instead of writing their own..
Scott Rudy, try telling your CIO why your programmers can't troublehoot "the deploy" to production after it worked on their local machines just fine. Or try getting in a weeks-long unproductive troubleshooting session with "the server guys" when neither of you knows how to do low-level networking troubleshooting, or try telling your CIO why when one system goes down, all the other systems go down too now that they're all calling its web services. There's a shareholder-value-focus if I've ever seen one. SHAZAM, YOU GOT SERVED...with business value talk! Feel the burn!
Peter, I can see you enjoy learning and that is a great quality. You obviously have a lot to learn outside of the technical realm, including concepts like shareholder value and humility. I have to wonder if your post is just troll BS, or you actually had a bad experience working for a company. If you did have a bad one, then I can assure you that your experience is not the only one and you should try to be more open to working with others. Based on your scenario this is what I would do based on my experience:
I would recommend termination for the rogue programmer that deployed a system in to production after testing it on his local machine, and proceeding to bring down several systems that were crucial to her business operations. Then I would back out the change and redesign it so I didn't have a single point of failure that would bring all the systems down (ESB, Queue, etc).
Once I fixed the designer's mistakes I would test it in an integration environment and a UAT environment with all the systems it would be touching, and ensure it had been properly load tested.
If at any point I had networking issues, I most certainly would not just say those "server guys" don't know anything and attack the problem by myself. I would work with teams in the enterprise to assist in troubleshooting. If they couldn't resolve it then I would lean on my partners to assist with their product specialists and then later address my staffing needs going forward.
Lastly, I certainly wouldn't run out and Tweet, "Look what I said to some dude", because you will end up destroying professional relationships and find it hard to get people to want to work with you. Good luck in your career.
Scott, if you meet me in person someday we can have a good laugh. We would get along famously.
I am not taking it too seriously and my serious point above is that, Ayende is probably aware of the business-focused- aspects of learning the basics and you seem to think since he didn't say it explicitly, he didn't take it into consideration.
My point was to say there are several business-impacting scenarios that would (and from my personal experience, HAVE and DO) arise from lack of understanding of those same basics.
Then I referenced the movie You Got Served and its many sequels.
Just curious Ayende; what would you consider to be your top (5?) reads as far as tech-related books go?
@Scott
You talk about maximizing shareholder value in the same breadth as recommending unknowledgable devs creating distributed systems with WCF - a config/designer-driven unnecessary complex, leaky abstraction that promotes poor remote web service practices and a friction-encumbered dev model.
Exactly how many alternate web service frameworks have you evaluated?
The number 1 trait of system rewrites I've been commissioned to replace is a result of un-knowlegeable devs trying to build distributed systems with over-architected 'Enterprise' frameworks like WCF. It ironically works and scales in the small, but fails when your system needs to evolve where its code-base size and complexity increases or systems load increases and you reaches your low scalability limits breaking the timeouts of their chatty N+1 RPC method calls because devs have been 'protected' from knowing that a network even exists. Since WCF/SOAP by-passes natural HTTP caching - most systems with performance / scalability issues will need a rewrite.
WCF wouldn't exist on any other platform, where frameworks survive based on merit and their value proposition alone - not because it's the only option hard-wired in VS.NET. Which will be once again be deprecated and replaced by Microsoft who will follow everyone else (years before them) into simpler, more streamlined HTTP/REST solutions. Once its been replaced the 'API knowledge' of these developers are effectively useless as they'll need to go back to the drawing board re-learning a fundamentally different API abstraction (like taking a UI from a drag n' drop dev). OTOH knowledgeable devs who understand everything that's going on under the hood already know what they're looking for and are just mapping new APIs to existing concepts.
Ayende is absolutely right here, its much more effective to spend time learning the fundamental concepts rather than waste time learning false API abstractions and its idiosynchroncies which provide the end-developer no value in better understanding the task at hand so they're better able to rational about the system their building.
WCF-only devs are going to be in the same boat as ASP.NET Web Forms devs who only know false, artificial server-side abstractions, useless once given a new framework - years behind everyone else who know how the web works.
Okay Dennis, so you don't like WCF. Let's take it out of the picture and replace it with another technology like NHibernate, which is an abstraction over a distributed system. I could make some of the same arguments you make about WCF about ORMs. Are you saying that developers can't develop systems using an ORM with a RDBMS if they don't learn the ins and outs of B-trees, relational algebra and TCP/IP? No, I am sure you are not going to state that. I don't need to know TCP/IP to avoid an N+1 situation. I need wisdom and experience to know that's not good.
My point is plenty of development can go on at a higher level of abstraction where the focus is on how the business runs. At the end of the day your customers don't care if you use TCP, HTTP, or UPS if they can get their job done. Every business is different, but in most cases a 5 second delay due to something like a missing entry in a routing table is a better issue to have than a transaction that results in breaking the integrity of your data.
@RobConnery's NDC presentation on ORMs just about conveys my thoughts on NHibernate and other heavyweight ORMs: 30 years of wasted DB Abstractions better replaced with a few lines of dynamic.
http://ndc2011.macsimum.no/mp4/Day2%20Thursday/Track1%201140-1240.mp4
As you might tell, I dislike heavy abstractions, code-gen, unclean models, XML config, etc. This post best describes how I would improve ASP.NET MVC which despite being modern still suffers some legacy ails.
http://www.servicestack.net/mvc-powerpack/
@Demis Bellot,
I have mixed feelings about that presentation of Rob Connery. He starts bashing all abstractions like NHibernate and then goes on by introducing his own abstraction again. That's just rediculous.
@Frank
Except its no abstraction at all, its just a in-line, debuggable DRY API over ADO.NET's raw data access interfaces, with none of the configuration, XML mappings, dynamic proxy classes, artificial abstractions (hiding DB existence with its own cruft), 3.5MB dll dependencies, blocking Startup times that NHibernate offers.
Honestly it puzzles me people can't tell the difference between the two.
NHibernate is on the decline and will eventually be superseded, and once again we'll see another heavy data access layer deprecated in time.
The only constant in 30 years of deprecated data access is SQL - the very thing everyone tries to hide even exists. Some have tried to do it with their own leaky artificial abstractions. I prefer the simpler approach of thin, DRY APIs over whatever we're trying to abstract - which in the case of ORMs is SQL.
Scott, If you can't understand what actually happen, how on earth do you except to be able to make reasoned decisions about your design and implementation?
Dave, Google skills are certainly valuable. But you need the base knowledge to actually be able to understand the answers you find. I have seen far to many people fall down into the "copy & paste" hell.
Dan, That probably deserve a post of its own, I'll try to write one soon
Scott, Anyone trying to use NHibernate without a good understanding on how actual databases work is going to end up in a world of pain. I say that as someone who has seen literally hundreds of applications written by people with various skill levels. And yes, being able to say how a specific query will be executed on the database matters. For example, if you don't understand B-Tree you may select a poor PK strategy. If you don't understand relational algebra, you model is going to be bad, etc.
Ayende, you need to find your comfort level with a level of abstraction and get moving on solving problems instead of wasting time with details that may never matter. You next post illustrates what is important, experience. It is your experience that defines you not your ability to study a manual and pass a test. The more projects you work on the better you will be positioned to make decisions moving forward.
Scott, I just gave you a list of reasons why you really care about the DB when using NHibernate, and why you care about TCP & HTTP when using WCF.
To put it simply, you can make do without that knowledge, but if you want to be able to build good software, that is essential.
Comment preview