Monday, January 21, 2013

Mental Hygiene for Software Engineers: Avoiding Micro Level Technical Debt

The essence of software engineering, in my opinion, can be captured in two words: managing complexity.   Perhaps it's not the same for others, but, for me, the most critical aspect of my job as a software engineer is to manage complexity.  Software is complex.  Far too complex for the all too human brain.  The human brain is not only limited by finite resources but it is subject to a variety of error prone behaviors that, while making us warmly human, are not necessarily a good fit for applications high levels of complexity and requirements for mathematical accuracy.  But this is old news, and the raison d'etre for software engineering as a field.

The essence of software engineering, in my opinion, can be captured in two words: managing complexity

But the focus of sofware engineering is on the code level and above.  Software design patterns, test driven methodologies, refactoring tactics, UML . . . these are all great tools that help achieve a cleaner, less complex software artifact.  But what about the lower level stuff?  No body looks much lower than the code, but the "physical" layer, if you will, consists of an individual engineer's minute to minute confrontation and resolution of tasks.  

Let's call this the micro level of software engineering.  Just like we should strive for clean code, we should also pay attention to these lower level tasks.  Most of the activity at this level is the personal domain of the individual engineer.  It's the stuff of how you apply your mental powers to the tasks at hand.  Instead of clean code, perhaps the goal at this level is clean mental processes.  Instead of hunting down code smells, we'll search for smells coming from our mental processes.  If we expect our code to be clean, we should start with our own mental hygiene.  

My assertion is that there are patterns and anti-patterns at the level of an individiual engineer's minute by minute processes that have impacts parallel to those on the code level.  Just as unclean, untested code can create a large technical debt that drags down efforts to maintain, modify and enhance the software, unclean workaday habits can lead to micro level technical debt that drags down the cognitive ability of the individual engineer.  Sure, we're all smart, but can any of us afford to be cavalier with the management of our mental capacities?  

Instead of hunting down code smells, we'll search for smells coming from our mental processes.  If we expect our code to be clean, we should start with our own mental hygiene.  

Let's consider a possible micro level anti-pattern: the micro level work around.  One of the most detrimental mental idiosyncrasies of software engineers is the pathological use of workarounds to avoid the micro level barriers that arise constantly during the work day.  I used to teach English and I like to draw a parallel to the habits of poor readers.  Many students, particularly those who struggle with reading comprehension to some degree, demonstrate several micro level reading behaviors that pretty much guarantee their failure.  One of the most glaring misteps is that poor readers often skip past words they don't understand.  They do this so much that it becomes a habitual, unconsious practice.  

The obvious best practice is to consult a dictionary.  But that takes time.  And it's only one word, out of pages of text, and who knows where the dictionary is anyway.  To the harried student, it seems that it would be impossible to complete the reading if one stopped to look up every new word.  But this just doesn't work.  Good reading takes advantage of the strong foundation of meaning given by previous sentences and paragraphs.  This semantic context is critical for comprehension of the sentences and paragraphs that follow.  If the reader has built a strong foundation from the preceding text, the meaning of an isolated unknown word might be easily guessed.  But, in truth, there will typically be another word and then another, and then another, and . . . soon, this literatary debt builds up, eroding the reader's comprehension further and further, leaving little context upon which to interpret even the sentences made of words the reader actually knows.

Just as unclean, untested code can create technical debt . . . the unclean workaday habits can lead to micro level technical debt that drags down the cognitive ability of the individual engineer.

This applies to the daily chores of a software engineering as well.  There's always something that isn't quite right, that you don't have the time to take care of correctly.  Perhaps it's the clumsy use of an ill-understood command.  Perhaps it's unwieldy permissions on a crucial set folders and files.  Perhaps it's a command that you type out many times a day rather than creating an alias.  Perhaps it's all of those crufty warnings in your build or in your application's logs.  These micro level barriers have many forms and arise constantly.  If left unattended, they quickly build up into your own personal technical debt.           

Let's consider a specific example.  I've watched more than one developer memorize phone directory amounts of raw IP numbers.  Sit and watch them work and you'll see them repeatedly type IP numbers as they ssh to a remote host, enter server addresses in browsers, connect to fileshares, anywhere and everywhere.  As amazing as this feat of memory is, you have to wonder what the cost is.  There's no getting around the fact that by memorizing these numbers you have not managed the complexity, you've beaten it back with brute force.  You've decided to allocate a portion of your mental capacity to a task that you could have solved more elegantly.

Worse yet, the whole effort that you've put into the brute force management of those IP numbers has ZERO re-use value.  It's a one off, developer specific throwaway.  Can a new developer to your team leverage your efforts?  No.  Do you personally even gain anything from the efforts?  Is there an item in your backlog that requires all developers to memorize the IP's of all developer environment servers?  By memorizing those four octets, you haven't reduced complexity of the raw IP's one bit. 

Just to state the obvious, the whole reason we strive to reduce the complexity of software is because, if unchecked, that complexity will consume 100% of our individual mental resources.  My assertion is that the same thing can happen to you from micro level technical debt.  I challenge you to confront each and every barrier that arises and defeat it.  Map those IP's to meaningful host names.  Create aliases for those commands that you constantly retype.  Declare a 30 minute moratorium on IM'ing!  Just clean up your mind!

I imagine the true blue multi-taskers are going to scoff.  For me personally, this is an essential part of being both a happy and a first rate engineer.  If you find it interesting, and wondering how to get started, I highly recommend checking out the pomodoro technique.  It's a systematic approach to the application of your mental powers.  

Thursday, April 12, 2012

Maven Does Not Suck . . . but the Maven Docs Do

I'm not going to go into the whole Maven debate, but suffice it to say that I'm a strong proponent of everything best practice, and, to me, Maven is an embodiment of best practice. By this I mean that Maven is built around a specific best practice build methodology. Note, I said a specific best practice build methodology. In the real world, there are more than a handful of build methodologies that could qualify for the best practice accolade, but Maven assumes a single one of them. This does not mean that the others are not good, it just means that if you use Maven, you going to need to buy-in to the conventions it assumes . . . or suffer. This is true for any Convention Over Configuration ( CoC ) tool, and Maven is pretty darn  CoC.

Maven, like all design patterns, is a reuseable solution to the process of building software

I think the occasionally discussed notion of Maven as a design pattern for builds is a powerful metaphor.  It's useful because it emphasises that Maven, like all design patterns, is a reuseable solution to the process of building software.  It's a best practice solution that has been refined by a community of smart people over years of heavy use.  The most obvious benefits of leveraging a design pattern for building software are the same as those for writing software.  Namely:

  • You get a bunch of functionality with out having to write it yourself
  • An engineer that understands the pattern as applied to one project, can instantly understand the pattern as applied to another project.

Nominally, the first bullet is about productivity and and the second is about simplicity. Obviously, everybody wants to be more productive, i.e. accomplishing more with less lines of code. But, I actually think the second point -- simplicity -- is far more important. In my opinion, the entire field of engineering boils down, most elegantly, to the concept of "managing complexity". By complexity, I refer directly to that headache you get when bombarded with piles of spaghetti code.  Design patterns help eliminate this intellectual discord by sealing off a big chunk of complexity in a higher level notation.  In case you've forgotten, this is what frees our minds up for the bigger and cooler tasks that inevitably reside on the next level.

It is this point of view that makes me rank learning a new project's ad hoc build to be one of the most annoying aspects of my profession. Even if an ant or make build is very cleanly implemented, follows a localized best practice, and automates a broad scope of the software lifecycle, it still punishes  new developers with a mountain of raw data, i.e. lines of scriptcode. Note, it's only the ad hoc-ness that is a problem here. This is certainly not a knock on these tools. ant in particular is very good at automating your tasks and providing a reusable set of build widgets. But it does nothing to provide a reusable solution to the entire process of building software, and, accordingly, it does nothing to ease a new developers on their road to comprehending the build.

it's the conventions that matter most with a CoC tool like Maven

So, as I see it, it's the conventions that matter most with a CoC tool like Maven.  You have to know and follow the assumed conventions in order to be successful with Maven. Projects that don't follow the conventions quickly run afoul of Maven. First, they struggle to implement their own build process with a tool that assumes a build process of it's own. It's easy to fall into being upset that you can't easily do what you've been doing, but the preceding paragraphs are meant to suggest that it's actually you who needs to change, at least if you plan to continue on with Maven. When you choose Maven, you need to accept the conventions. I you can't, I suggest you stick with Ant, which is flexible enough to meet you on your terms. Just remember that you are losing the ability to leverage the design pattern aspect of Maven to manage the complexity of your build. And if you think your build doesn't have complexity issues, ask your self these questions:

  • Can every engineer on our team easily build all the components of our software system?
  • Do our engineers have the confidence to modify build scripts without angst?
  • Do our engineers flee the room when someone is needed to address a build problem?

So, if you're with me so far, you'd probably agree that following the conventions assumed by Maven is a critical prerequisite for entering Maven nirvana. And this is what leads me to the conclude that the Maven docs suck.  They are not only inadequate, but perhaps detrimental; they mostly document the configuration while utterly failing on the critical topic of conventions.  The emphasis on configuration, which I assume is largely by accident, leads newbies into thinking it's okay, and perhaps even  normal, to configure Maven.

The Maven documentation is not only inadequate, but perhaps detrimental; it mostly documents the configuration while utterly failing on the critical topic of conventions.

By documentation, I mostly mean all that stuff you find when you visit the Maven or Codehaus plugin pages. For instance, consider the extremely core maven-assembly-plugin.  Browse through the docs on the Maven site and you'll find that it's almost entirely about configuration. The problem, as I've stated and restated, is that you don't really want to configure Maven; you want to follow the conventions. Configuration should be only an option of last resort.

plugin puts things and then the next plugin can't find that stuff.  Use a profile to tell Maven where to find something, and then nothing else can find that thing without the profile.  Configuring Maven gets you into a bit of a configuration feedback loop, and geometric growth of configuration does not lend itself to pom readability.  Even if you can get Maven to do what you need by configuring it to death, you quickly get an incomprehensible build.  

Use the configuration to change where one plugin puts things and then the next plugin can't find that stuff.

So, avoid configuration!  Stick instead to the conventional path. Your engineers will know and love their build, and you will easily leverage the many benefits offered by the Maven ecosystem -- from the rich plugin libraries to the repository servers and build servers.  

But how does one go about learning the Maven conventions?  It's all about community. Luckily, it's a pretty friendly community.  Here are some of the most important resources that I use when trying to determine how things should be done in Maven.

Additionally, in an effort to be a friendly community member, I'm using this blog entry as an introduction to a series of Maven entries.  Each of these entries will outline important Maven conventions. I'll detail the convention as well as offer example poms.  So, keep in touch if you want to learn about Maven conventions.  

Sunday, March 27, 2011

Agile Development: A Reminder

Just a minor rant, I mean, reminder . . .

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

This is the core of the Agile Manifesto, of course, reprinted here as a gentle reminder.