Every red-faced technical debate that I’ve ever gotten into has almost always devolved into one topic:  is this solution simple or is it complex?  And yet, there’s one alternative that we always forget about: simplism.  Simplism is simplicity’s evil twin, and it can be difficult to tell the two apart.  The difference between the two is that simplicity will grow up to be a successful adult who will make you proud, while simplism will grow up to be in jail, doing drugs, and always borrowing money from you.

Ok, so that analogy is a little bit hokey.  Let’s try looking at the dictionary definitions:

simple - easy to understand, deal with, use, etc.: a simple matter;simple tools.

simplistic – oversimplifying complex problems; making unrealistically simple judgments or analyses

When you do something simplistic, you are essentially sticking your head in the sand. You’re implementing a solution that meets the requirements of some problem that’s much simpler than the one you’re presently facing or will be facing in the future. And it’s always the latter part of the last sentence that gets you. Most people are smart enough to know when a solution doesn’t meet their present needs. But trying to find a solution that will be flexible enough to meet whatever random occurrence should come up in the future is tough.

Some people will tell you to stop right there. You don’t know what’s going to happen in the future, so don’t worry about it. But I think that the sign of truly elegant code is that it can be made to do all sorts of things its original author never dreamed of. There’s no need to contort it to do crazy things because it just works. And before you write that off as over-engineering, stop to think of the bigger picture. Remember: business requirements have a habit of changing suddenly in the most unforeseen ways. To get things done, sometimes you have to be willing to not get them done right this minute. As the Zen of Python says:

Now is better than never.
Although never is often better than right now.

So how do you keep things simple without making them simplistic?

I’ve found exactly one method that reliably makes software simpler without making it simplistic: make the software do less stuff. From a strictly technical standpoint, this is a surefire way to make software that is simple and not simplistic. Of course the business side will insist that you meet the requirements and then some. And they have good reason. I only said that this makes things simpler from a technical perspective. Technical simplicity might be business simplism.

But 9 times out of 10, there’s a way to satisfy both sides if they’re willing to compromise. There’s something to be said for the 90% solution. It does 90% of the things business people want and is 90% perfect for the programmers. The 10% you cut out might not sound like much, but if you cut out the right 10%, you’ve earned your salary along with one of your coworkers'.

So what’s the moral of this story? Before you decide on the solution that seems “simplest”, stop and think. Is it really simple, or is it simplistic?

After rereading my last post, I realized that it didn’t really paint introverts in the best light. It makes introverts seem selfish. That’s actually a good thing because that’s how introversion tends to look to extraverts. But that’s not really accurate.

Here’s the funny thing about personality: disputes tend to happen between people when they have opposite ways to do the same thing. Whereas extraverts respect people by integrating their thoughts into the whole, introverts respect people by leaving them alone. They view trying to merge peoples' thoughts into one big whole as an attempt to “water them down”. Thus, introverts will constantly try to differentiate themselves from the common point of view.

That said, introverts would do well to heed Derek Sivers’s advice that sometimes Persistence is Polite. People aren’t usually as put off by an introvert’s “intrusion” as they imagine them to be.

Thus far, I’ve only focused on applying the concept of introversion and extraversion to how we make decisions. But they also affect perception.

Let’s look at Joel Spolsky. If you look at Spolsky’s famous “Don’t rewrite your software” blog post, you can see that Joel is obviously extraverted when it comes to making decisions. It might be easier to see this if you realize that psychologists refer to extraverted attitudes as “objective” and introverted ones as “subjective”. In other words, extraverts tend to look for the “one true” plan or way of seeing things while introverts will be focused on trying to find a plan or truth that works for them. You can see Spolsky’s extraversion in decision-making when he says things like:

They did it by making the single worst strategic mistake that any software company can make

or

The old mantra build one to throw away is dangerous when applied to large scale commercial applications.

or

But throwing away the whole program is a dangerous folly

Heck, even the title of the blog post (“Things you should never do”) is extraverted. These are all couched in very logical language, but taken together you can see an overarching theme: “There’s one true way to write software, and the only way to figure it out is for everyone to add their little piece of the truth. Here’s mine.”

However if you look, it’s clear that Spolsky also has an introverted side. Consider the following phrases:

The reason is that they think the old code is a mess.

or

The consensus seems to be that the old Netscape code base was really bad. Well, it might have been bad, but, you know what? It worked pretty darn well on an awful lot of real world computer systems.

or

“Well,” they say, “look at this function. It is two pages long! None of this stuff belongs in there! I don’t know what half of these API calls are for.”

You can see that Spolsky has an introverted way of perceiving the world that complements his extraverted decision-making. The hallmark of the introverted attitude is the desire to separate oneself from others. The overarching theme of these phrases is “they see the world like this, but I see the world like this”, which is introverted. That the others are wrong is just a side effect.

Again, even though I’ve pointed out Spolsky’s introverted side and extraverted side, remember that this is still oversimplified. I’ve pointed out what I see as his primary ways of looking at things, but you can see others in the rest of his text.

So how can an extravert get along with an introvert?

1. Agree to disagree. Extraverts tend to dislike ending a conversation without everyone coming to a consensus. Unless the matter is completely trivial, introverts dislike ending a conversation without having differentiated themselves from the other person somehow. If possible, try to find a happy medium. End the conversation by pointing out things you both agree on (carefully: make sure to couch this with phrases like “It sounds to me like you agree with me on x, y, and z”), but acknowledge that there are areas where you both disagree and that’s ok.
2. Realize that when an introvert is quiet, that doesn’t mean they don’t respect you. If you need feedback from them, try to do so in a way that is respectful to them. For instance, say something like “Hey, whenever you have a moment can we discuss x?”
3. Don’t go too far in accepting their worldview. Some people (both introverted and extraverted) might try to take advantage of your desire to connect with other people. Always be skeptical of someone whose judgement of you is dependent upon you giving them (or not giving them) something they want.

One of Carl Jung’s great contributions to the field of psychology is the idea of the archetype.  There is a boatload of psychology books on this subject out there if you have a lot of free time to spend deciphering psychological jargon or just happen to also have a PhD in psychology (in which case this blog post wouldn’t be useful to you anyway).  Here’s the thing though:  you already know what an archetype is.  You just haven’t put a name to it.

Think of the characters that are in every story.  There’s always a hero and a villain.  They also have a supporting cast of other characters.  You have the parent (metaphorically speaking – they don’t have to be someone’s actual parent), a good natured mentor who protects people (think Obi-Wan from Star Wars).  Then there’s the senex:  the wise man who treats his proteges like crap because that’s the only way to learn (think of the sensei in any martial arts movie).  You’ve also got the Puer, the good natured, innocent child-like character (just about every role Adam Sandler plays).  And there’s also the trickster:  the child-like character who causes trouble (Kevin from Home Alone). These are all archetypes.

Not only do we see these characters in fiction, but we also see them in real life too.  Here’s where things get interesting though.  We all share these archetypes, but we put different people in each role. Your hero might be my senex.   Your puer might be my villain.

The reason?  We place people in the role of these archetypes based on ourselves, not other people.  You might feel that a part of your personality is particularly child-like and associate it with the puer when you see it in other people.  Seeing through this illusion can be difficult.  However, it’s usually very liberating to see through this.  While others are always looking for heroes and villains and such, you get a chance to see people as they actually are.

It’s no coincidence that each of these archetypes has a “good” and “bad” version, either.  The “good” version is usually more conscious. The “bad” version usually comes from the unconscious.

It might be easiest to see this with an example. Compare Joel Spolsky’s blog post about Duct Tape programmers to Michael Lopp’s essay on Free Electron programmers.

Ostensibly, these programmers are both really productive. But the similarity ends there. Spolsky’s Duct Tape programmer is someone who doesn’t go with the latest fad. They’d rather just get things done than spend their time worrying about what’s fashionable. They also don’t spend too much time worrying about making their code perfect because they want to ship. Lopp’s Free Electron programmer on the other hand “defines the bleeding edge”. They can get things done, but you have to be careful. They’ll rewrite your database layer from scratch for that .1 release.

What’s most interesting is that they both refer to the same person (unless Lopp means someone else when he mentions Netscape’s Free Electron — UPDATE: apparently he did, jwz claims to not be able to ride a unicycle). Now Lopp actually worked with Zawinski so his version might be more accurate, but I suspect that both are projecting themselves onto him at some level.

In other words, these kinds of writings tell us more about the author than they tell us about what makes a programmer good. This is what we do when we talk about our Hero: emphasize its strengths while we gloss over its weaknesses. This makes sense if you realize that the Hero essentially represents our ego. Any attack to the Hero is an attack on the ego.

It seems to me that hacker culture seems to be focused too much on doing lately. There’s nothing wrong with doing things. In fact, I like doing things quite a bit. However, not a week goes by when I don’t see another blog post about how to be more productive and get more done.

I sometimes wonder if hackers are the new teenage girls in a certain sense. Teenage girls are constantly bombarded with images of women who are so beautiful in ways they’ll never be. Of course, we’ve all heard of the negative effects this has. Girls become anorexic and go on unhealthy diets just so they can be beautiful. They’re too caught up in being like supermodels that they have difficulty accepting that they really don’t have to be supermodels to be beautiful.

Us hackers are a lot like that. Hacker news is constantly bombarded with stories about hackers getting amazing things done. Everytime I’ve been to a hacker meetup of some kind, the first and only question is always some variant of “So what are you working on?” Of course the answer is supposed to be either you’re putting in hours and hours on your startup or you have a day job and another “job” that is your real passion.

I’m not saying that there’s anything wrong with some constructive encouragement. However, I think we’re collectively going down the wrong path. The path of having spent our youths solving a ton of problems that people probably won’t even care about 10 years from now anyway.

So in the vein of the “Cult of Done Manifesto”, I present the “Cult of Not Done Manifesto”. This something I’ve been unconsciously writing my entire life because that’s all I’ve had:

1. There are two stages of being: not done and less not done.
2. Accept that no programmer has ever died from having their project be in the not done status for too long.
3. Sometimes good things take time.
4. Slack off. Spend time you should be spending being productive reading a blog or conversing with a friend.
5. Your project will be there for you tomorrow. And the next day. If it isn’t, are you sure it was such a big deal anyway?
6. Ask yourself a question: is it really that important? Will peoples' lives be better 10 years from now because of what you’re doing? Chances are good that the answers to these two questions are the same.
7. Just make sure to at least get something done. The Cult of Not Done isn’t an excuse to be a worthless bum.

Metaclasses have a reputation for being among the blackest of Python’s black magic. As it turns out, they’re actually pretty simple, but they require you to understand some concepts. There are two ways to learn metaclasses in Python:

1. Skip the details and get straight to a useful implementation of a metaclass.
2. Understand the core concepts behind metaclasses.

Number 1 is useful, but will only get you so far. If you really want to understand metaclasses, you need to take approach number 2.

With that in mind, I’m going to start with the basics of how classes are constructed in Python. Let’s consider the following class:

class SomeClass(object):
     x = 1
     y = 2

If you’re ready to learn about metaclasses, the above statement shouldn’t require much thought to understand. But let’s stop to think about it a bit anyway. In Python, everything is an object. Therefore, classes are also objects. But if SomeClass is an object, it must be an instance of some class. Let’s find out what that class is:

>>> type(SomeClass)
<type 'type'>

So apparently, it’s an instance of type. We just saw the most common usage of type: to query an object’s type. But have you ever read the help on that function?

>>> help(type)

Help on class type in module __builtin__:

class type(object)
 |  type(object) -> the object's type
 |  type(name, bases, dict) -> a new type
 ...

So it turns out that type is not only a builtin function, it’s also a class! How do we instantiate an instance of type? You’ve already seen the most obvious way: using a class statement. But did you know that you can create a class using type?

SomeClass = type('SomeClass', (object,), {'x' : 1, 'y' : 2})

For all intents and purposes, the above statement is equivalent to the prior definition of SomeClass. But this way is ugly and isn’t used very commonly. That said, we’ve demonstrated something: type isn’t just any class. It’s the class of classes. It’s a metaclass.

Let’s go a step further though. How does the compiler generate the dictionary that’s the third argument to type? As it turns out, classes have something in common with functions: they have local namespaces. You might have seen the locals function used like this:

def some_func():
    x = 1
    y = 2
    print locals()

If you execute this, the output is this:

>>> some_func()
{'y': 2, 'x': 1}

If classes have their own namespaces, then they must also be able to use the locals function as well:

>>> class SomeClass(object):
...     x = 1
...     y = 2
...     print locals()
...
{'y': 2, 'x': 1, '__module__': '__main__'}

The only difference here is that the namespace is generated at import time and passed into the type function.

So this is all interesting, but we haven’t really seen anything terribly useful yet. Let’s go further. We’ve seen that type is the class of classes. But if type is a class, then we must be able to subclass it. There are a number of reasons you might want to do this. For instance, you may sometimes want to attach a property to a class rather than to an instance. Let’s do just that:

class SomeMetaClass(type):
    @property
    def z(self):
        print 'In class property z'
        return self.x + self.y

>>> SomeClass = SomeMetaClass('SomeClass', (object,), {'x' : 1, 'y' : 2})
>>> SomeClass.z
In class property z
3

But we don’t want to use this same ugly notation for creating SomeClass. Python provides syntactic sugar for this. We can instead define SomeClass like this:

class SomeClass(object):
    __metaclass__ = SomeMetaClass
    x = 1
    y = 2

A more common use of metaclasses is to create a class constructor. Let’s attach z to the class directly rather than defining a property:

>>> class SomeMetaClass(type):
...         def __init__(self, name, bases, dict):
...             self.z = self.x + self.y
...
>>> class SomeClass(object):
...         __metaclass__ = SomeMetaClass
...         x = 1
...         y = 2
...
>>> SomeClass.z
3

As we can see, we’ve defined a constructor for SomeClass. Now let’s go a bit further. What if we want to change the base class of SomeClass? That can be done, but we have to use a __new__ method. I’m going to presume that you know a bit about __new__ methods. If you don’t, you might want to read up on them.

>>> class SomeMetaClass(type):
...     def __new__(cls, name, bases, cls_dict):
...         new_cls = type.__new__(cls, name, (dict,), cls_dict)
...         return new_cls
...
>>> class SomeClass(object):
...     __metaclass__ = SomeMetaClass
...     x = 1
...     y = 2
...
>>> x = SomeClass()
>>> x['foo'] = 'bar'
>>> x
{'foo': 'bar'}

That should hopefully give you an idea of what metaclasses are and how to use them. If you’re even more lost than you were before, don’t worry. This is just one of those things that requires an “aha!” moment. You might also want to check out Michael Foord’s great Meta-classes Made Easy for a different perspective.

If you’re just starting with Celery right now, you’re probably a bit confused. That’s not because celery is doing anything wrong. In fact, celery does a very good job of abstracting out the lower-level stuff so you can focus just on writing tasks. You don’t need to know very much about how any of the messaging systems you’re using will work. However, to truly understand celery, you need to know a bit about how it uses messaging and where it fits in your technology stack. This is my attempt to teach you the things you need to know about the subject to be able to make everything work.

Messaging

At the very bottom of celery’s technology stack is your messaging system, or Message Oriented Middleware in enterprise-speak. As of this writing, there are a couple of standards out there in this market:

• AMQP – A binary protocol that focuses on performance and features.
• STOMP – A text-based format that focuses on simplicity and ease of use.

Of course, there are a lot more players out there than just this. But these are the two protocols that are the most important to celery.

Now, a protocol is totally useless without software that actually implements it. In the case of AMQP, the most popular implementation seems to be RabbitMQ. The popular implementation of STOMP seems to be Apache ActiveMQ.

Carrot

A good analogy that I think most people can wrap their heads around is the SQL database. STOMP and AMQP are like SQL, while RabbitMQ and ActiveMQ are like Oracle and SQL Server. Any one who has had to write software that works with more than one type of database knows how challenging this can be. Sure, it’s easy to issue SQL commands directly when you just support one type of database, but what happens when you need to support another? One possible solution is to use an ORM. By abstracting out the lower-level stuff, you make your code more portable.

The first thing most ORMs do is provide an abstraction to write SQL queries. For instance, if I want to write a LIMIT query for SQL Server, I would do something like this:

SELECT TOP (10) x FROM some_table

Oracle’s query would look something like this:

SELECT x FROM some_table WHERE row_num < 10

These are different queries, but they are both doing the same basic thing. That’s why SQLAlchemy allows you to write the query like this:

select([some_table.x], limit=10)

This is the functionality that carrot provides. Although most messaging systems are fundamentally different in a lot of ways, there are certain operations that every platform has some version of. For example, sending a message in STOMP would look like this:

SEND
destination:/queue/a

hello
^@

AMQP’s version is binary, but would look something like this in text format:

basic.publish "hello" some_exchange a

Since we don’t want to worry too much about these protocols at a low level, carrot creates a Publisher class with a “send” message.

Celery

Carrot makes it so that we can forget about a lot of the lower-level stuff, but it doesn’t save us from the fact that we’re still working with a messaging protocol (albeit a higher-level one). Going back to the ORM analogy, we can see the same thing happening: we need a layer of abstraction to make dealing with different implementations of SQL easier, but we don’t want to write SQL. We want to write Python (or whatever your language of choice is).

Thus, ORMs will add another layer of abstraction. Wouldn’t it be nice if we could just treat a database row as a Python object? Or, in the case of task execution, wouldn’t it be nice if we could just treat a task as a Python function? This is where celery comes in. See, we could run tasks like this:

1. Process A wants to run task “foo.bar”
2. Process A puts a message in queue saying “run foo.bar”
3. Process B sees this message and starts on it
4. When done, Process B replies to Process A with the status.
5. Process A acknowledges this message and uses the return result.

Rather than having to code all the details of the messaging process, celery allows us to just create a Python function “foo.bar” that will do the above for us. Thus, we can execute tasks asynchronously without requiring that people reading our code know everything about our messaging backend.

Hopefully, this gives you a high-level overview of how celery is working behind the scenes. There are a lot of details that I’ve left out, but hopefully this provides you with enough knowledge that you can figure the rest out.

If there's one part of a programmer's professional development that is harder than it seems, it's starting a side project. I, for instance, love to program. The funny thing is, that programming is the easiest part of programming (those of you who are true programmers will know what I mean). The hardest part is starting to program and finishing your work.

That said, I do have a few bits of advice for programmers who want to start a side project. If you follow my advice, you probably won't write the next big thing, nor will you have a side project that will stand out particularly well on your resume. But you'll probably write something that will make you a better programmer, and that's the point, isn't it?

Here's what I've learned about starting a side project:

#1 - Have a good time

This is the first point on here for a reason. And I should hope it's self explanatory. However, note that there are good reasons for making tradeoffs here. I'll note them below, but there are some things that shouldn't be tradeoffs. For instance, don't make tradeoffs based on how it will look on your resume or what the latest and greatest technology is (not that either of these can't be fun).

#2 - Learn something

I've known some programmers who live by the mantra "I'll learn it later". Here's the problem: later never comes. You're just going to have to accept the fact that you as a professional need to have some responsibility for your own development. Fortunately, a side project is just the way to take on that learning.

#3 - Make it timeless

Here's where you may have to make tradeoffs in terms of fun. The most fun things are usually a hassle to maintain. For instance, suppose you write a new twitter client. I can guarantee you that at some point in time, the twitter people will change their API in such a way that will break your code. Granted, this isn't a big deal if you put a lot of time into it. But that's the problem: no matter what you do, there's going to come a point in time in your life that you have something else to spend your time on or otherwise just won't want to deal with your side project. That's normal. What you want is something that you can come back to after that time is up.

This is part of the reason why I chose pysistence for my project. Aside from keeping up with language changes, it's doubtful that functional data structures are ever going to change such that my code will break.

#4 - It doesn't have to be code

I'd probably say that the blog you're reading right now is my most rewarding side project. Don't underestimate the value of doing things that aren't coding. It's fashionable to talk about how bad it is to be "all talk and no game", but the fact of the matter is that reading and writing about programming helps you to solidify some of the ideas that may be forming in your head.

#5 - Make it something you'll use

This is the last point, but that doesn't mean it's not important. If you write something fun, you'll abandon it as soon as it stops being fun. If you write something useful, you'll have a reason to keep coming back!

I just came across a blog post that outlines 5 rules for writing good code.  I agree with them for the most part.  But this subject is extremely subjective and will vary from person to person.  Therefore, I'd like to write up my own rules for writing good code.