Programming’s dirty word

Us programmers tend to be a logical, pragmatic bunch. We’re good at choosing the option that is strictly speaking the most practical. And yet, I have yet to hear a programmer ask the question “Will anyone want to work with this?”

And there’s the dirty word: want. Many times, we get so caught up in finding the way that will be the most efficient that we forget why we became programmers. I don’t know about you, the reader, but I became a programmer because it’s fun. The moment I stop enjoying what I do is the moment I stop being a programmer and start being a paid coding monkey.

And here’s the kicker: this rabid focus that we have on the practical isn’t practical. It’s quite idealistic in fact. Looking back, the best projects that I’ve completed aren’t the ones where I had an easy, efficient way out. In fact, they’re frequently projects where I get to try something new out or get my hands dirty discovering better ways to solve old problems. In short, as long as you don’t get too distracted from the task at hand, wants are practical.

Now, I already know what counterarguments I’m going to receive from this post, so let me take a moment to address them:

  • You can’t always get what you want. And you would be absolutely right to make this point. No matter what you do, you’re always going to run into a situation where you have to do something you don’t want to do. However, you don’t completely abandon the idea of life just because you’re going to die at some point. Instead, you focus on living as long and enjoyable a life as you can.
  • You’re putting your own desires ahead of your team’s. I’m certainly not arguing that any individual programmer needs to be able to sabotage the rest of the team just because they want something. It is extremely important that you take what others want into consideration in addition to what you want.
  • Programmers need to focus on building a product, not the code. This sounds like something a Product Manager or startup founder would say. And I would agree with it to a certain point. It’s important to work on a product that’s important to you. However, I’m not a Product Manager or startup founder. I’m a programmer, and product design really makes up about 10% of my overall job (and that’s being generous). The rest of that time is spent coding. And programmers shouldn’t have to apologize for wanting to make that time as pleasant as possible.
  • You’re not being pragmatic. You’re right, I’m not being completely pragmatic, and neither should you. The ideal solution is one that is pragmatic and enjoyable. Given a choice between a pragmatic solution and fun one, you should always choose the pragmatic one. After all, a fun but completely impractical idea will never see the light of day. However, things are rarely so black and white. You have to learn to balance tradeoffs. Such as the tradeoff between practicality and enjoyment.

I suppose the main thing that I’m trying to say here is that what you want is an important consideration in addition to what you need (despite what the Rolling Stones would have you believe).

Persistence is a subset of immutability

Whenever I tell someone I’m working on a persistent data structure
library for Python, I almost always get the same response:
“Persistent means that something is stored across process boundaries.
I think you mean to say immutable data structures.”  In fairness, this
response is understandable.  But it’s not quite right.  I’m going to
address both of the points in the above statement.

Persistent means that something is stored across process boundaries

Did you know that letter can mean a note you send through the mail or a thing you use to make up words?  Did you know that programming can mean writing computer programs or mathematical optimization?  I’m sure if I sat down long enough, I could think of more examples.  The point is that the language we speak is full of ambiguities and whatnot.

Before we consider the computer science meaning of persistence, let’s
look at the common meaning.  If we look at the word persistent in
Merriam-Webster, the first two definitions we run into are:

1 : existing for a long or longer than usual time or continuously: as
a : retained beyond the usual period  b : continuing without change in
function or structure 

The meaning Python programmers tend to think of is the definition in
part a.  That is, a persistent data structure is stored beyond the
usual period (the process’s lifespan).  However, when I talk about a
persistent data structure, I tend to mean the definition in part b.

Going with definition b, we can see that lists for example aren’t persistent:

  1. >>> a = [1,2,3]
  2. >>> b = a
  3. >>> b.append(4)
  4. >>> a
  5. [1, 2, 3, 4]

 

Tuples, however, are a different matter:

  1. >>> a = (1,2,3)
  2. >>> b = a + (4,)
  3. >>> b
  4. (1, 2, 3, 4)
  5. >>> a
  6. (1, 2, 3)

 

Ah, but here’s where the second part of this post comes in.

I think you mean to say immutable data structures

In the case of the tuple, you are correct to say that a tuple is
immutable.  And I’m correct in saying that a tuple is persistent.
Just like I’d be right to say that I own a car while someone else
claims that I own an automobile.  However in both cases, they can’t
quite be used interchangeably.  After all, an automobile isn’t
necessarily a car.  It can be a truck or a van or an SUV.

In the same sense, a persistent data structure is effectively
immutable.  But an immutable data structure isn’t necessarily
persistent.  Let’s consider the definition of both of these words from
wikipedia:

Immutable object

In object-oriented and functional programming, an immutable object is 

an object whose state cannot be modified after it is created. 
Persistent data structure

In computing, a persistent data structure is a data structure which 

always preserves the previous version of itself when it is modified; 

such data structures are effectively immutable, as their operations

do not (visibly) update the structure in-place, but instead always

yield a new updated structure. 
These definitions have a lot of common ground, but there is some
difference between them.  Let’s consider a data structure that is
immutable but not persistent.  Prepare yourself.  This will be
complex.  Ready?

  1. >>> 1
  2. 1

 

Has your mind been blown yet?

In this sense, the number 1 is definitely immutable.  You can’t change
it.  It’s always the number 1.  However, it’s not persistent.  How do
you update the number 1?  No matter what you do, it will always be the
number 1.  Sure, you can get 3 by adding it to 2.  But in a
mathematical sense, that’s not really changing the number 1.

In this sense, the number 1 is atomic.  It simply doesn’t make sense
to modify it.  Heck, you can’t even copy it:

  1. >>> from copy import copy
  2. >>> a = 1
  3. >>> a is copy(a)
  4. True

Conclusion

With this semantic difference in mind, remember that it’s just that:
a semantic difference.  Call them Bob if you want to.  However, bear
in mind that when someone uses the word persistence in this way,
they’re not being inaccurate.

REALLY Caring for Your Introvert

After rereading my last post, I realized that it didn’t really paint introverts in the best light. It makes introverts seem selfish. That’s actually a good thing because that’s how introversion tends to look to extraverts. But that’s not really accurate.

Here’s the funny thing about personality: disputes tend to happen between people when they have opposite ways to do the same thing. Whereas extraverts respect people by integrating their thoughts into the whole, introverts respect people by leaving them alone. They view trying to merge peoples’ thoughts into one big whole as an attempt to “water them down”. Thus, introverts will constantly try to differentiate themselves from the common point of view.

That said, introverts would do well to heed Derek Sivers’s advice that sometimes Persistence is Polite. People aren’t usually as put off by an introvert’s “intrusion” as they imagine them to be.

Thus far, I’ve only focused on applying the concept of introversion and extraversion to how we make decisions. But they also affect perception.

Let’s look at Joel Spolsky. If you look at Spolsky’s famous “Don’t rewrite your software” blog post, you can see that Joel is obviously extraverted when it comes to making decisions. It might be easier to see this if you realize that psychologists refer to extraverted attitudes as “objective” and introverted ones as “subjective”. In other words, extraverts tend to look for the “one true” plan or way of seeing things while introverts will be focused on trying to find a plan or truth that works for them. You can see Spolsky’s extraversion in decision-making when he says things like:

They did it by making the single worst strategic mistake that any software company can make

or

The old mantra build one to throw away is dangerous when applied to large scale commercial applications.

or

But throwing away the whole program is a dangerous folly

Heck, even the title of the blog post (“Things you should never do”) is extraverted. These are all couched in very logical language, but taken together you can see an overarching theme: “There’s one true way to write software, and the only way to figure it out is for everyone to add their little piece of the truth. Here’s mine.”

However if you look, it’s clear that Spolsky also has an introverted side. Consider the following phrases:

The reason is that they think the old code is a mess.

or

The consensus seems to be that the old Netscape code base was really bad. Well, it might have been bad, but, you know what? It worked pretty darn well on an awful lot of real world computer systems.

or

“Well,” they say, “look at this function. It is two pages long! None of this stuff belongs in there! I don’t know what half of these API calls are for.”

You can see that Spolsky has an introverted way of perceiving the world that complements his extraverted decision-making. The hallmark of the introverted attitude is the desire to separate oneself from others. The overarching theme of these phrases is “they see the world like this, but I see the world like this”, which is introverted. That the others are wrong is just a side effect.

Again, even though I’ve pointed out Spolsky’s introverted side and extraverted side, remember that this is still oversimplified. I’ve pointed out what I see as his primary ways of looking at things, but you can see others in the rest of his text.

So how can an extravert get along with an introvert?

  1. Agree to disagree. Extraverts tend to dislike ending a conversation without everyone coming to a consensus. Unless the matter is completely trivial, introverts dislike ending a conversation without having differentiated themselves from the other person somehow. If possible, try to find a happy medium. End the conversation by pointing out things you both agree on (carefully: make sure to couch this with phrases like “It sounds to me like you agree with me on x, y, and z”), but acknowledge that there are areas where you both disagree and that’s ok.
  2. Realize that when an introvert is quiet, that doesn’t mean they don’t respect you. If you need feedback from them, try to do so in a way that is respectful to them. For instance, say something like “Hey, whenever you have a moment can we discuss x?”
  3. Don’t go too far in accepting their worldview. Some people (both introverted and extraverted) might try to take advantage of your desire to connect with other people. Always be skeptical of someone whose judgement of you is dependent upon you giving them (or not giving them) something they want.

Simplism

Every red-faced technical debate that I’ve ever gotten into has almost always devolved into one topic:  is this solution simple or is it complex?  And yet, there’s one alternative that we always forget about: simplism.  Simplism is simplicity’s evil twin, and it can be difficult to tell the two apart.  The difference between the two is that simplicity will grow up to be a successful adult who will make you proud, while simplism will grow up to be in jail, doing drugs, and always borrowing money from you.

Ok, so that analogy is a little bit hokey.  Let’s try looking at the dictionary definitions:

simple – easy to understand, deal with, use, etc.: a simple matter;simple tools.

simplistic – oversimplifying complex problems; making unrealistically simple judgments or analyses

When you do something simplistic, you are essentially sticking your head in the sand. You’re implementing a solution that meets the requirements of some problem that’s much simpler than the one you’re presently facing or will be facing in the future. And it’s always the latter part of the last sentence that gets you. Most people are smart enough to know when a solution doesn’t meet their present needs. But trying to find a solution that will be flexible enough to meet whatever random occurrence should come up in the future is tough.

Some people will tell you to stop right there. You don’t know what’s going to happen in the future, so don’t worry about it. But I think that the sign of truly elegant code is that it can be made to do all sorts of things its original author never dreamed of. There’s no need to contort it to do crazy things because it just works. And before you write that off as over-engineering, stop to think of the bigger picture. Remember: business requirements have a habit of changing suddenly in the most unforeseen ways. To get things done, sometimes you have to be willing to not get them done right this minute. As the Zen of Python says:

Now is better than never.
Although never is often better than right now.

So how do you keep things simple without making them simplistic?

I’ve found exactly one method that reliably makes software simpler without making it simplistic: make the software do less stuff. From a strictly technical standpoint, this is a surefire way to make software that is simple and not simplistic. Of course the business side will insist that you meet the requirements and then some. And they have good reason. I only said that this makes things simpler from a technical perspective. Technical simplicity might be business simplism.

But 9 times out of 10, there’s a way to satisfy both sides if they’re willing to compromise. There’s something to be said for the 90% solution. It does 90% of the things business people want and is 90% perfect for the programmers. The 10% you cut out might not sound like much, but if you cut out the right 10%, you’ve earned your salary along with one of your coworkers’.

So what’s the moral of this story? Before you decide on the solution that seems “simplest”, stop and think. Is it really simple, or is it simplistic?

How to have a side project

If there’s one part of a programmer’s professional development that is harder than it seems, it’s starting a side project. I, for instance, love to program. The funny thing is, that programming is the easiest part of programming (those of you who are true programmers will know what I mean). The hardest part is starting to program and finishing your work.

That said, I do have a few bits of advice for programmers who want to start a side project. If you follow my advice, you probably won’t write the next big thing, nor will you have a side project that will stand out particularly well on your resume. But you’ll probably write something that will make you a better programmer, and that’s the point, isn’t it?

Here’s what I’ve learned about starting a side project:

#1 – Have a good time

This is the first point on here for a reason. And I should hope it’s self explanatory. However, note that there are good reasons for making tradeoffs here. I’ll note them below, but there are some things that shouldn’t be tradeoffs. For instance, don’t make tradeoffs based on how it will look on your resume or what the latest and greatest technology is (not that either of these can’t be fun).

#2 – Learn something

I’ve known some programmers who live by the mantra “I’ll learn it later”. Here’s the problem: later never comes. You’re just going to have to accept the fact that you as a professional need to have some responsibility for your own development. Fortunately, a side project is just the way to take on that learning.

#3 – Make it timeless

Here’s where you may have to make tradeoffs in terms of fun. The most fun things are usually a hassle to maintain. For instance, suppose you write a new twitter client. I can guarantee you that at some point in time, the twitter people will change their API in such a way that will break your code. Granted, this isn’t a big deal if you put a lot of time into it. But that’s the problem: no matter what you do, there’s going to come a point in time in your life that you have something else to spend your time on or otherwise just won’t want to deal with your side project. That’s normal. What you want is something that you can come back to after that time is up.

This is part of the reason why I chose pysistence for my project. Aside from keeping up with language changes, it’s doubtful that functional data structures are ever going to change such that my code will break.

#4 – It doesn’t have to be code

I’d probably say that the blog you’re reading right now is my most rewarding side project. Don’t underestimate the value of doing things that aren’t coding. It’s fashionable to talk about how bad it is to be “all talk and no game”, but the fact of the matter is that reading and writing about programming helps you to solidify some of the ideas that may be forming in your head.

#5 – Make it something you’ll use

This is the last point, but that doesn’t mean it’s not important. If you write something fun, you’ll abandon it as soon as it stops being fun. If you write something useful, you’ll have a reason to keep coming back!

The Rise of SYDNI, or YAGNI is Only About Problems, Not Solutions

I’ve got a new programming methodology to propose. I call it SYDNI
(Sometimes You Do Need It). It is a response to the problems that I
see with YAGNI. In fairness, I don’t dislike YAGNI. In fact, I agree
with it 100% (well, maybe 95%). But to truly appreciate it, you need
a bit of context.

On YAGNI

I’ve almost started thinking of YAGNI almost as a recursive way of
thinking. That is to say that I’ve begun to think of YAGNI as being
something that uses itself to implement itself. Allow me to explain.

What is YAGNI?

YAGNI stands for “you ain’t gonna need it.” I don’t want to make this post
an in-depth discussion of what YAGNI actually is, so click the
Wikipedia link if you aren’t familiar with YAGNI. The important thing
to take away from reading about YAGNI is that it’s saying that you
shouldn’t implement functionality if you don’t need it.

What YAGNI ISN’T

YAGNI sounds like a pretty straightforward way of thinking. And in a
lot of ways it is. But it’s more nuanced than one may think at first.
The “recursive” element of YAGNI that I speak of above is that YAGNI
(in my opinion) is a very specific solution to a very specific
problem, and that problem is over-engineering.

And YAGNI does its job well (especially in the context of Test Driven
Development). I tend to find myself throwing out a lot less code when
using YAGNI.

A lot of people take YAGNI to mean that the simplest solution is
always the best. That isn’t the case. Or at a very minimum, that
shouldn’t be the case. There’s a key thing about simplicity
that should be understood: it’s defined by the problem, not the
solution. This is key to understanding why YAGNI is so useful. Once
you’ve gotten to the point of choosing a solution, YAGNI is no help to
you. Thus, you have to use YAGNI to choose problems, not solutions.

You’re not in school anymore

In school, things are always so simple. You’re assigned a problem.
And you’re given a grade based on how well you solved that problem.
The real world is more complex.

You see, people too often forget that software developers don’t just
define solutions to problems. After all, aren’t all feature requests
nothing more than a statement of a problem? And isn’t choosing
software features a decision about what problems you will solve?

However, once you’ve chosen a problem to solve, there’s still the
issue of how to solve it.

Sometimes You Do Need It

In solving a problem, YAGNI’s usefulness starts to fade. It does have
some importance. You do have to make sure your solution is solving
the problem you set out to solve. However, beyond that, YAGNI just
doesn’t apply. In fact, it is likely harmful. That’s where SYDNI
comes in. Although SYDNI’s name is something of a jab at YAGNI, the
principle itself isn’t. Instead, SYDNI can be thought of as a
complement to YAGNI. A yin to YAGNI’s yang (alliteration for the
win).

Oftentimes, thoughts may enter your head that start with something
like “we’ll never need…” or “this will never have to…”. This kind
of thinking is helpful when choosing the problem to solve. However,
it’s destructive when choosing a solution. In a couple of years,
there is only one thing that will be certain about the software you’re
writing: it will be different. And it will be different in ways you
can’t have predicted or imagined. If you’re using YAGNI
appropriately, you’re choosing the easiest problems to solve.
However, at least a few of these problems come out of left field.

Therefore, I would put SYDNI this way: ideally, a piece of software
will be no more simple or complex than the problem it is trying to
solve. Therefore, there is danger not only in solutions that are
overly complicated, but there is also danger in solutions that are
overly simple.

This leads to another conclusion: if SYDNI is followed appropriately,
the complexity of your source code is a direct measure of how complex
the problems it is solving are. The reverse is true as well. The
complexity of the problems you’re solving is a direct measure of how
complex your source code will be.

But I don’t live in an ideal world!

The key hole in SYDNI is the word “ideally”. Unfortunately, some
problems just don’t have perfectly compatible solutions. Therefore, a
key decision to be made is whether it is better to err on the side of
over-engineering or under-engineering a solution. We are now delving
into the realm of many disputes between programmers. Many people
(mis)educated on the arts of YAGNI will say that it is always better
to tend towards under-engineering. If this were true, YAGNI wouldn’t
be as useful to as many people as it has been.

Even more unfortunately, there is no “one size fits all” answer of
whether it is better to over-engineer or under-engineer. It is highly
situational and care must be taken to arrive at the appropriate
solution. If you don’t believe me, consider the following two
questions:

  1. Which life support machine would you rather be hooked up to?
    •  A machine whose software developers always did the simplest thing possible
    •  A machine whose software developers went out of their way to anticipate possible problems and planned for each of them
  2. Which one-page web app do you feel would be easiest to maintain?
    • An application that is implemented as two or three source files and a few database tables
    • An application with a highly normalized database, highly modular source, and great flexibility

 

I should hope that the answer to number 1 is obvious. And why it is
the correct answer should also be obvious: if you missed a particular
contingency, people can die. Thus, it makes sense to err on the side
of over-engineering.

But number 2 is a little bit less obvious (and maybe more debatable).
However, I would err on the side of under-engineering. After all, no
matter what changes come up, a one-page web app is still a one-page
web app. The worst case is that the app would be rewritten from
scratch. That’s not to say that you need to throw caution into the
wind and ignore normal good practice. Rather, it’s saying that it’s
not really a good idea to stress much over how maintainable that
application is.

Therefore, when deciding on a solution, there are two things that need
to be decided upon beforehand:

1. How complex the problem is.
2. Whether under-engineering is more harmful than over-engineering.

Once you get those two things squared away, it should be easy to get
an idea of how complex the solution should be.

The devil’s in the details

There are two schools of thought in the programming world:

  1. Explicit is better than implicit (configuration over convention)
  2. A developer should only have to program the unconventional aspects of a program (convention over configuration).

We’ll call #1 the Python school and #2 the Ruby school. In fact, I
would argue that this is an issue that’s at the core of whether code
is considered “Pythonic” or “Rubyic” (I doubt the last one is a word).

So which school of thought is right? I personally think they both
are. It doesn’t really take a whole lot to demonstrate that the
Python school of thought isn’t always right. Think about it. Did you
know that the Python runtime has a component that goes around deleting
objects from memory totally implicitly? How unpythonic is that?

The Ruby school of thought takes a bit more work though. After all,
if it’s unconventional, why should you have to configure it? Of
course, the problem here is in defining “conventional”. What’s
conventional to me is likely unconventional to others. And what’s
conventional to others could be unconventional to me.

I wish I had more advice on how to reconcile these two schools of
thought. The truth is that I struggle with them daily. But I think
having an intuition about this is the dividing line between
“experienced programmer” and “newb”. After all, if programming were
merely about “make everything explicit” or “make everything implicit”,
any idiot could do it.

I think this is also the core skill for writing readable code. You
need to determine what details are relevant to each piece of code.
Whatever the case, you need to make a conscious decision as to what
details shine through and what details you obscure. Because if these
things happen on accident, they’re almost guaranteed to be wrong.

My five rules for writing good code

I just came across a blog post that outlines 5 rules for writing good code.  I agree with them for the most part.  But this subject is extremely subjective and will vary from person to person.  Therefore, I’d like to write up my own rules for writing good code.

Keep it simple

This is the YAGNI rule.
There are often times when we want to try to solve problems we don’t have.  You must resist this urge.  It’s far easier to make simple code more complex than the other way around.  This is usually more of a challenge than it looks.  It’s a sign of good code that you constantly find yourself saying “any idiot could have put this together”.  Reality is that idiots only write simple code when the problem is easy or when they get lucky.

…but not simplistic

You can call this the SYDNI rule.
Albert Einstein said it best when he said “Everything should be made as simple as possible, but not simpler.”  I’ve seen too many “simple” hacks that ended up causing more of a maintenance problem than it would have been just to write something more complex.  As good a thing as simplicity is, you need to be realistic.  Don’t try to make a simple solution match a complex problem.

Abstraction is your friend

So what do you do in those situations where you need to use a complex solution?  Do you give up all hope and just write some horrible piece of crap?  No, you find a way to make that complexity easier to manage.  This is where abstraction comes in handy.
For instance, Lisp introduced the concept that “it’s all data”. This makes it easier to understand things.  If you want to know what something is, you already know that it’s data of some kind.  What is a function?  Data.  What is a list?  Data.  What is code?  Data.  This has a profound effect upon your ability to understand things.

Follow guidelines

Jeff Atwood would call this following the instructions on the paint can.  We tend to sneer at the idea of “best practices”, but reality is that they’re necessary.  There are a lot of problems out there that people have already dealt with and solved.  Why waste time not learning from others’ mistakes.

…but don’t worship them

As the old cliche tells us, rules were made to be broken.  Some of the most well known design patterns break the guidelines and have some of the worst code smells.  In fact, sometimes the guidelines conflict with each other.  Thus, don’t blindly follow guidelines without knowing their purpose.  Instead, understand the rules, know when to follow them and when following them is the greater of two evils.

How Celery, Carrot, and your messaging stack work

If you’re just starting with Celery right now, you’re probably a bit confused. That’s not because celery is doing anything wrong. In fact, celery does a very good job of abstracting out the lower-level stuff so you can focus just on writing tasks. You don’t need to know very much about how any of the messaging systems you’re using will work. However, to truly understand celery, you need to know a bit about how it uses messaging and where it fits in your technology stack. This is my attempt to teach you the things you need to know about the subject to be able to make everything work.

Messaging

At the very bottom of celery’s technology stack is your messaging system, or Message Oriented Middleware in enterprise-speak. As of this writing, there are a couple of standards out there in this market:

  • AMQP – A binary protocol that focuses on performance and features.
  • STOMP – A text-based format that focuses on simplicity and ease of use.

Of course, there are a lot more players out there than just this. But these are the two protocols that are the most important to celery.

Now, a protocol is totally useless without software that actually implements it. In the case of AMQP, the most popular implementation seems to be RabbitMQ. The popular implementation of STOMP seems to be Apache ActiveMQ.

Carrot

A good analogy that I think most people can wrap their heads around is the SQL database. STOMP and AMQP are like SQL, while RabbitMQ and ActiveMQ are like Oracle and SQL Server. Any one who has had to write software that works with more than one type of database knows how challenging this can be. Sure, it’s easy to issue SQL commands directly when you just support one type of database, but what happens when you need to support another? One possible solution is to use an ORM. By abstracting out the lower-level stuff, you make your code more portable.

The first thing most ORMs do is provide an abstraction to write SQL queries. For instance, if I want to write a LIMIT query for SQL Server, I would do something like this:

SELECT TOP (10) x FROM some_table

Oracle’s query would look something like this:

SELECT x FROM some_table WHERE row_num < 10

These are different queries, but they are both doing the same basic thing. That’s why SQLAlchemy allows you to write the query like this:

select([some_table.x], limit=10)

This is the functionality that carrot provides. Although most messaging systems are fundamentally different in a lot of ways, there are certain operations that every platform has some version of. For example, sending a message in STOMP would look like this:

SEND
destination:/queue/a

hello
^@

AMQP’s version is binary, but would look something like this in text format:

basic.publish "hello" some_exchange a

Since we don’t want to worry too much about these protocols at a low level, carrot creates a Publisher class with a “send” message.

Celery

Carrot makes it so that we can forget about a lot of the lower-level stuff, but it doesn’t save us from the fact that we’re still working with a messaging protocol (albeit a higher-level one). Going back to the ORM analogy, we can see the same thing happening: we need a layer of abstraction to make dealing with different implementations of SQL easier, but we don’t want to write SQL. We want to write Python (or whatever your language of choice is).

Thus, ORMs will add another layer of abstraction. Wouldn’t it be nice if we could just treat a database row as a Python object? Or, in the case of task execution, wouldn’t it be nice if we could just treat a task as a Python function? This is where celery comes in. See, we could run tasks like this:

  1. Process A wants to run task “foo.bar”
  2. Process A puts a message in queue saying “run foo.bar”
  3. Process B sees this message and starts on it
  4. When done, Process B replies to Process A with the status.
  5. Process A acknowledges this message and uses the return result.

Rather than having to code all the details of the messaging process, celery allows us to just create a Python function “foo.bar” that will do the above for us. Thus, we can execute tasks asynchronously without requiring that people reading our code know everything about our messaging backend.

Hopefully, this gives you a high-level overview of how celery is working behind the scenes. There are a lot of details that I’ve left out, but hopefully this provides you with enough knowledge that you can figure the rest out.

Tail Recursion in Python using Pysistence

A topic that occasionally comes up in Python development is that of tail recursion. Many functional programmers want to see tail recursion elimination added to the Python language. According to Guido, that ain’t gonna happen. And to be fair, I agree. Tail recursion can be tricky not only for new programmers, but for old timers as well.

However, that doesn’t mean that we need to give up on the concept altogether. This is a problem that is hardly new. Functional languages have been implementing tail recursion in environments hostile to it for a while now using a trampoline approach. Let’s see how the newest version of pysistence implements that algorithm:

def trampoline(func, *args, **kwargs):
    """Calls func with the given arguments.  If func returns 
       another function, it will call that function and repeat
       this process until a non-callable is returned."""
    result = func(*args, **kwargs)
    while callable(result):
        result = result()
    return result

This makes it much easier to implement things functionally (using pysistence’s functional lists):

def iter_to_plist(seq):
    seq = iter(seq)
    def inner(accum):
        try:
            return partial(inner, accum.cons(seq.next()))
        except StopIteration:
            return accum
    return inner(pysistence.make_list())

>>> trampoline(iter_to_plist, xrange(1000))
PList([999, 998, 997, 996, 995, 994, 993, 992, 991, 990, 989, 988, 987, 986, 985

That was more work than writing this in a language that does tail recursion automagically. But it wasn’t too bad now was it? Ultimately, I think this approach works for a few reasons:

  1. It’s explicit. The user is well aware of what’s happening because they’re returning a callable.
  2. It makes functional programming more natural. Instead of using true recursion and risking blowing the stack or converting this into a loop and continually reassigning to a variable, you can make the algorithm work without side effects.
  3. It lets Python stay imperative.

For me personally, this is a great set of arguments. I love Python and I love functional programming. The more functional programming I can do in Python, the better. But there are very few things in programming that are free, right? Here are some of the disadvantages:

  1. It’s ugly. I don’t deny this. But this alone is not an argument against it. If you are the kind of person who wants nothing but beautiful code, I’d argue that you’ve chosen the wrong language. Python tends to be beautiful when possible, but ugly when it has to be. Besides that this is the best you’re going to get without modifying the language itself or adding macros.
  2. It isn’t very performant. Insert your favorite quote about not needing to be blazing fast, just good enough here. Besides that, it can be optimized in C if need be.
  3. Who needs more ways to do functional programming? I’m not going to join any flamewars on this one. But I will point you to the advantages of functional programming in Python’s own functional programming HOWTO.