Jasim A Basheer

Notes on Legible Code, for both humans and AI

2026-04-28T06:30:00Z

Short set of notes on how code should be written, so it is habitable by humans and legible to AI. It is drawn from practitioners, but based on what I've come to want in my projects. It is written as a guide for humans to read, and as a skill for AI coding agents to use.

An invitation to ReasonML

2017-05-10T10:40:00Z

Peter Deustch once quipped that if you get the data structures and their invariants right, most of the code will just kind of write itself. If I had to pick just one nugget of learning from my career in programming so far, that would be it.

The effects of a poorly designed model cascades across the codebase, forcing us to litter it with special cases. That is probably why Linus Torvalds once claimed that the difference between a bad programmer and a good one is whether they consider their code or their data structures more important. Terry Crowley's article Education of a Programmer makes a similar observation about how important the data flow is in the design of a system:

Jon Devaan used to say “design the data, not the code". This also generally means when looking at the structure of a system, I’m less interested in seeing how the code interacts — I want to see how the data interacts and flows. If someone tries to explain a system by describing the code structure and does not understand the rate and volume of data flow, they do not understand the system.

I decided to learn ReasonML because I wanted to work in a language that revolved around data. I had become dissatisfied with programming in dynamic languages like Ruby and Javascript, and all my code started feeling fragile and rickety as soon as it was written. To build large systems, I needed to have a crisp, confident knowledge of the shape and flow of data across the codebase at all times. But the unpredictability inherent in dynamic languages made this impossible.

The evolution of view templating in 9 easy steps

2014-12-25T14:40:00Z

If you squint hard enough, Greenspun's tenth rule says:

Any sufficiently complicated templating language contains an ad hoc, informally-specified, bug-ridden, slow implementation of a turing complete programming language.

Hindsight is a wonderful thing. Let's see what it teaches us about the current state of the V in MVC:

1. It all started with the render method.

def greet
  name = params[:name]
  render text: "Hello #{name}!"
end

2. Then the text grew rich and spouted HTML tags:

def greet
  name = params[:name]
  render text: "<html><h1>Hello #{name}!</h1></html>"
end

3. It became unwieldy, and multi-line string was born.

def greet
  name = params[:name]
  render text: %{
    <html>
      <h1>Hello #{name}!</h1>
    </html>
  }
end

4. There was more to say and more to do. We had to render a todo list, for which the code acquired loops:

def greet
  name = params[:name]
  todos = ["wake up", "step into the hamster wheel"]

  todosView = todos.map.with_index {|todo, index|
    %{
      <p>
        #{index+1} -
        #{todo}
      </p>
    }
  }.join("\n")

  render text: %{
    <html>
      <h1>Hello #{name}!</h1>
      #{todosView}
    </html>
  }
end

5. But this violated certain principles, to rectify which the first templates were born. Instead of muddling the code with string munging, we split the view into a separate template file, and passed just the required data into it:

def greet
  name = params[:name]
  todos = ["wake up", "step into the hamster wheel"]

  render file: "greeting.html.erb", locals: {
    name: name,
    todos: todos
  }
end

6. The template, mostly static, contained a few dynamic portions for which it had to do things like asking questions (conditionals), repetition (loops), and re-use (partials). This meant interspersing the content with code:

<html>
  <h1>
    Hello <%= "#{name} !" if name %>
  </h1>

  <% todos.each_with_index do |todo, index| %>
    <p>
      <%= index+1 %> -
      <%= todo %>
    </p>
  <% end %>
</html>

7. But everyone knew that mixing logic and view was wrong. Logic-less templates were born to get rid of this uneasy co-existence. They absorbed the imperative ugliness of code into declarative directives. Purity!

Things started to get weird just about then:

<html>
  <h1>
    Hello {{name }} {% raw %}
    <span ng-if="name">{{name}} !</span> {% endraw %}
  </h1>

  <p ng-repeat="todo in todos">{% raw %}
    {{ $index+1 }} -
    {{ todo }}  {% endraw %}
  </p>
</html>

8. This got the ball rolling. Templates went on an acquisition spree. It gets expressions and filters:

<input ng-model="searchText">

<p ng-repeat="todo in todos | filter:search"> {% raw %}
  {{ $index+1 }} -
  {{ todo }} {% endraw %}
</p>

9. There is a limit to the expressive powers of a language if it doesn't let the user build their own abstractions. Logic-less templates compensates for this by opting-in the abstractions of its host language:

{% raw %}
{{#list people}}
  {{firstName}} {{lastName}}
{{/list}}
{% endraw %}

Handlebars.registerHelper('list', function(items, options) {
  var out = "<ul>";

  for(var i=0, l=items.length; i<l; i++) {
    out = out + "<li>" + options.fn(items[i]) + "</li>";
  }

  return out + "</ul>";
});

And thus we come full circle. What started out as logic-less templates became template programming languages, replete with expressions, conditionals, loops, scoping rules, interop with a host language, in short, an ad-hoc bug-ridden implementation of a programming language.

View Templates began as a way to separate content from code. This used to work reasonably well in the past when most of the web was just content. The occasional need for programmable expression was easily satisfied by string interpolation. I'm writing this very post in a content-primary format (HTML with a few Jekyll helpers). It is a good compromise.

This paradigm however doesn't suit webapps. A website is a programmatically enhanced view, but a webapp is a programmatically constructed view. Construction calls for abstractions. We can either port back programming abstractions in the form of helpers and directives into templates, giving rise to unwiedly templating languages with custom syntax and innumerable quirks, or we could simply use code, the best tool for the job.

So how did view templates become so pervasive? I can count a few reasons:

The web was historically content oriented for which view templates were a natural medium.
It is easier for designers and front-end developers who solely write HTML+CSS to work on a view template than code.
Code was not suited for view templating because it couldn't differentiate between strings and XML view templates.

The HTML snippet embedded in titleView = "<h1>#{"Hello!".upcase}</h1>" can only be parsed as a string, and not as an XML view. This means we cannot get any tooling support from the editor. Moreover, building large strings using concatenation is unwieldy.

But it is easy to parse view templates. Consider the following ERB template:
<h1><%= "Hello!".upcase %></h1>. The point at which the template ends and code begins is very clear cut. This enables programming editors to parse and and provide tooling around it - linting, syntax highlight, auto-indent, autocomplete, jump to etc.

Out of the above three points, both the first and last ones no longer hold. The web has become dynamic and content takes the back-stage. As to tooling, Facebook made code friendly to XML in 2010 with the release of XHP. With XHP, developers could embed XML into PHP with no extra syntax. This idea was brought to front-end development by JSX+React. Building applications with React's component model is very pleasant, the resulting view code is better organized and more maintainable (I talk about how React makes the front-end a fully-programmable system in this post), and the tooling support for JSX on IntelliJ IDEs is excellent.

With JSX, we are able to do templates better than they could. We can mix and match code and views and build well organized front-end code without having to reinvent programming constructs into yet another templating language.

On React

2014-11-18T14:40:00Z

React makes the front-end a fully programmable system

This is why programmers are going gaga over it. And that it takes just two and a half days to grok.

A React component is a self-contained unit that encapsulates both the view and view logic. This abstraction is fundamental to a front-end application, just like classes are to an object-oriented app, or tables to a database app.

Here is how a hierarchy of React components for an imaginary Project Management page would look like:

ProjectPage
- StoryWall title="Current"
  - Story title="I should be able to see the Project"
- StoryWall title="Backlog"
  - Story title="I should be able to see the Walls"
  - Story title="I should be able to see each Story"

These components map directly to both your front-end code and the actual UI, making the project highly explorable.

View as a first-class citizen

In React, the views are written in Javascript. With that kind of power, this is the kind of things you get to write:

var story = {
  title: "I should be able to see the project",
  isCompleted: true
};

function storyView(story) {
  var title = story.title
  var isCompleted = story.isCompleted ? "completed" : "pending";

  return (
    <div className="Story">
      <h3>{title}</h3>
      <span>({isCompleted})</span>
    </div>
  );
}

If that was crazy, then look at how easily you can render an array of stories using _.map:

var wall = {
  title: "Current",
  stories: [story]
}

function wallView(wall) {
  var stories = wall.stories;
  var title = wall.title

  var storiesView = _.map(stories, storyView);

  return (
    <div className="StoryWall">
      <h2>{title}</h2>
      <div className="StoryWall-stories">
        {storiesView}
      </div>
    </div>
  );
}

Contrast this with traditional HTML based templating: they reinvent conditionals and other control structures, and the only way to reuse code is string munging through partials. Helpers and directives are high ceremony and can't be used as often as they should be. Add to that there are a whole slew of these languages that are conceptually similar but has the overhead of different syntaxes.

React makes the view layer fully programmable and thus elevates them to be first-class citizens of the codebase. This means you can can use variables and methods and control structures and all the other programming abstractions to write and refactor beautifully organized view code.

The React component syntax

The simplest React components are state-less. All they do is interpolate parameters into the view template. In this capacity it works as a functionally pure templating function, like Handlebars or Mustache.

React components are built using React.createClass({}). The API is minimal: it just needs a render method that returns a view.

Let us build Story, our first component, using the view code in the first example.

var Story = React.createClass({
  render: function() {
    var story = this.props.story

    var title = story.title
    var isCompleted = story.isCompleted ? "completed" : "pending";

    return (
      <div className="Story">
        <h3>{title}</h3>
        <span>({isCompleted})</span>
      </div>
    );
  }
});

Once you define a component using React.createClass, you render it using React.render. The first argument to React.render is the React view that you want to render, and the second is the DOM element into which you want to mount it. Here, let us render the component:

var story = { title: "I should be able to see a Story",
              isCompleted: true }

React.render(
  <Story story={story} />,
  document.getElementById('example1-Story')
);

And the result looks so:

Any parameter that you pass to a React component (like story in the above code) is available in the hash this.props inside the component. React considers props to be immutable, and so should you.

React and CSS - take one get one free

CSS used to be the one part of an otherwise clean codebase that first turns into a ball of mud. Turns out the trick to good CSS is to avoid cascade, the C in CSS. Cascade makes the style bleed from one unconnected part of the codebase to another, often in hard to predict ways. Modern CSS conventions - OOCSS, SMACSS, BEM etc. are all about containing this by isolating one component from the other.

However, each of these frameworks use their own terminology: BEM is about identifying reusable 'blocks', while SMACSS calls them 'modules'. But if you use React, there is nothing new to learn - you are already extracting reusable components naturally, and if your CSS adequately mirrors them, you are getting the benefits of well-factored code in your CSS for free. This is exactly what we did with our code examples: the CSS class names are the same as the component names. Naming is already hard enough; you shouldn't have to do it twice.

Any object oriented CSS framework should work well with React. I have recently started using SUIT CSS which works especially well with component-based UI frameworks. To begin with, these three conventions will take you far:

ComponentName--modifierName	eg: Notification--error
ComponentName-descendentName	eg: StoryWall-stories
ComponentName.is-stateOfComponent	eg: Story.is-active

Composing components

Now let us build the StoryWall component - it is also similar to the plain view code we first wrote, except that this time we wrap it as a React component.

var StoryWall = React.createClass({
  render: function() {
    var wall = this.props.wall;

    var stories = wall.stories;
    var title = wall.title

    var storiesView = _.map(stories, function (story) {
      return (<Story story={story}/>);
    });

    return (
      <div className="StoryWall">
        <h2>{title}</h2>
        <div className="StoryWall-stories">
          {storiesView}
        </div>
      </div>
    );
  }
});

We'll render it with some data:

var currentWall = {
  title: "Backlog",
  stories: [
    {
      title: "I should be able to see a Wall",
      isCompleted: true
    },
    {
      title: "I should be able to see the Project",
      isCompleted: false
    }
  ]
}

React.render(
  <StoryWall wall={currentWall} />,
  document.getElementById('example2-StoryWall')
);

In action:

Here we used the Story component inside StoryWall as seamlessly as we would use a normal HTML tag. There was nothing extra to be done. This is the other aspect of React that makes it highly programmable: components are almost as fluid as functions; they are parameterized, and are easy to both compose and reuse. There are no arcane incantations to be initiated into.

What next?

React is built on simple concepts and there is very little magic going on - as soon as you understand how to do something, you also understand how it works behind the scenes. React also favors the explicit is better than implicit philosophy, even if it comes with the overhead of extra code at times. Becoming productive with the library involve just a few days of learning curve, after which you are building and reusing components with ease.

If you are a back-end developer with a bad taste for Javascript, you'll be surprised how React and a sane collections library like Underscore can make the language pleasant to use. Quoting a convert:

React completely changed the way I thought about rich clients. It took me a little longer than 2 days, I went into React with an absolute hatred of anything Javascript and came out 2 weeks later loving it. There is something to be said for a library that can wash away over 10 years of built up anger for a particular technology. It's not just another revamped library thrown into the frying pan, it's a complete rethink of how rich clients can be developed almost effortlessly with code that anyone can read. When I got into react, all I could think was "yes, this makes sense, this is how it should have been all along." Moving into the future I expect most libraries will unify under the concepts introduced by React in the same way that most web applications frameworks are different flavors of MVC.

- as response to you'll never go back to anything else

If you work with Rails, there is good news: it is far easier to get started using React in Rails than it is to setup an independant Javascript project with Gulp, watchify, reactify et al. Just drop browserify-rails and react-rails and you are good to go.

Here are a bunch of areas that you'll cover as you start using React for real-world applications:

building stateful components and managing UI state
explicit communication between React components using callbacks
using the key attribute to help React differentiate between sibling elements
component lifecycle events, so that you can use external libraries like jQuery plugins
making use of the node.js ecosystem through npm and browserify

The official React documentation has a comprehensive tutorial that covers all these topics and more, including in-depth documentation on the working of the library itself. You should also watch Pete Hunt's excellent introduction to the React way of thinking to get a firm grip on the thinking behind React. I also recommend Ryan Florence's React Training lessons - the best resource on the web that covers real-world React use cases.

With React, becoming a front-end developer is easier than it has ever been before. You can start using it in your projects today - the component based approach means it does not need a big-bang adoption. If you have a rich widget that you'd otherwise build using jQuery and or Backbone, give React a go instead. You would be surprised at how fast you are able to write clean Javascript UI code, and how happy it makes you.

Review: Growing Rails Applications in Practice

2014-09-21T05:50:00Z

If you are a Rails developer with a few years under your belt and struggle with the occasional fat model, unwieldy controller and go berserk when you see the horrible mess of CSS we make, this book is for you.

It is published directly by the authors on LeanPub. Here is the link: Growing Rails Applications in Practice.

What is it about?

Rails gives our programs an implicit shape: ActiveRecord models hold domain logic, controllers orchestrate communication with the outside world, views hold forms, helpers can talk to views, and then there are CSS files. The amount of organization is minimal and works during the early duration of a project, but over time we find the control of the project slipping from our hands: the models become too fat, there is callback hell, the controllers seem to do too many things, and the CSS is a complete mess.

We develop coping mechanisms as we encounter them and over time a good developer will end up with a toolbox to manage this complexity. The authors of this book, Henning Koch and Thomas Eisenbarth, both from Makandra, the makers of Rails-LTS and Makandra Cards, have distilled the toolbox they built over years of working with Rails into this book.

They talk about namespacing - an under-appreciated technique in the Rails world, how to tease out PORO objects from ActiveRecord models, how to write clean, consistent controllers, and among others, how to use BEM to structure our CSS. Some of it reinforces existing wisdom about Rails code organization, and some of it taught me new tricks to manage large Rails projects.

Why is it relevant?

This is a common complaint from developers working on long-lived Rails projects: the codebase is too big, a tangled web of dependencies that are hard to reason about. And far too often the proposed solution is to split the large project into smaller ones. Wise advice, but it can be an extreme remedy.

Monolithic applications can go out of hand quickly if we don't pay attention. The reasons can be many: the Active Record pattern, Ruby's dynamic nature, and the inherent issues with monolithic systems. But when done right, monolithic applications can scale in both project complexity and team-size quite a bit before they need to be cut up into a finer grained architecture.

This book teaches how to do exactly that: building well-engineered monolithic systems that can be a delight to work on. Everything is described in the context of Rails, but the principles can apply to other programming situations as well.

Check out the book here: Growing Rails Application in Practice.

Tracking time

2014-08-19T05:00:00Z

What percentage of your daily working hours do you think you actually work?

I sometimes ask executives who pride themselves on their memory to put down their guess as to how they spend their own time. Then I lock these guesses away for a few weeks or months. In the meantime, the executives run an actual time record on themselves. There is never much resemblance between the way these men thought they used their time and their actual records.

-- The Effective Executive, Peter Drucker.

I can confirm from experience that this is as true for programmers as it was for Drucker's executives. We spend a lot lesser time in productive work than what we imagine we do. Sometimes we manage to get good work done only for half the time we think we did. If you aren't tracking your time yet, try it and see whether this holds.

A few months back I started weighing myself once every day. I didn't hit a gym nor did I enter a diet plan, I did nothing apart from being conscious of my weight on a daily basis. But over time I dropped a few kilograms and it has remained there since.

Measuring and tracking a metric - be it our weight, our expenses, or in this case how we spend time, is the first step to optimizing it. And often, just measuring alone can do the trick.

The Pomodoro technique is famous among programmers as an effective way to keep ourselves fresh and productive. It divides your day into intervals of 25-minute work followed by a 5-minute break. The crux of Pomodoro however is not the specific time intervals you choose, nor that you take a break often, but the fact that a timer is running when you work. A timer focuses you like nothing else can. A quick look at the email or a peek at your Twitter feed are out of question while you are timing yourself. Simple but pretty effective.

Software estimation for great fun and profit

2014-08-14T13:35:00Z

Have you seen a well-oiled agile team go about their work? It is a pleasure to watch. They have a rhythm for software delivery ‐ the programmers work in a sustainable pace and they meet business goals. I think this is because agile project management does two things right: story points for relative sizing and predicting future performance by using past averages (velocity and volatility). These two methods account for the fact that individual estimates are fallible and work around it.

But not all teams are agile. One reason is that customers don't understand how unpredictable software development can be.

"when will you ship?" asks the customer.

The requirements are a bunch of implied expectations scattered over many email threads. Do we fall into this trap?

"That is a trick question", says you. "Imagine that you decide to take a hike on the coast from San Francisco to Los Angeles..."

But you are interrupted.

"Hold on a sec, I'm not hiking anywhere! Just give me a date, okay?"

Thus begins a long conversation, the first, in a series of long conversations.

The other reason for projects to not use common-sense project management is that they simply can't. They may be in constant flux to allow any rhythm to take hold and their timelines too short to even start calculating a reliable average velocity. Real world, things hairy.

We have to up our game to work well in those environments. Without the luxury of sensible agile practices, we will have to get better at estimating individual tasks. The challenge then is to deliver software predictably without a working process to do so. It is an interesting challenge, and can even come with a few benefits. In this post, I'll argue that done right, estimating individual tasks well is a great productivity booster and makes software development a bit more fun.

But first, we have to understand what we are up against.

Estimates are commitments to ourselves

Is there a wall-poster which says estimates are not commitments? We should get one. We should read it in the mornings and evenings, before food and after food. Because the first casualties of an estimate are us, the developers.

You are working on a story that you had estimated to be a 1-pointer. It typically takes you a day to finish a point, but this one is taking longer. Anxiety creeps in, but you don't recognize it yet: this story is not supposed to take this long, ergo, I'm too slow.

We become wedded to the estimate, an arbitrary number ‐ a guess made from incomplete data, and it worries us when reality doesn't match up. This is "anchoring".

a cognitive bias that describes the common human tendency to rely too heavily on the first piece of information offered (the "anchor") when making decisions.

Various studies have shown that anchoring is very difficult to avoid. For example, in one study students were given anchors that were obviously wrong. They were asked whether Mahatma Gandhi died before or after age 9, or before or after age 140. Clearly neither of these anchors can be correct, but the two groups still guessed significantly differently (average age of 50 vs. average age of 67)

-- Anchoring, Wikipedia.

Dan Ariely has run significant experiments on Anchoring, about which he talks in his engaging book Predictably Irrational.

The anxiety induced by anchoring is not an articulated thought. It is a nagging thread in the back of the mind chipping into the joy of our work.

But it is a silly game that the mind plays and we can eradicate it by simply being aware of it. That is both easy and hard. Easy because once recognized, the worry is disposed off easily, and hard because mindfulness require an awful lot of practice.

We could have produced an accurate estimate in the first place and avoided this anxiety. But before we get to that, we have to look at what estimates do to our customers.

Estimates are commitments to our customers

Let me illustrate. You are asked for an estimate on when a big feature can be released, and the following happens:

You think: The stories total to around 45 points. Our average weekly velocity is 16 and volatility is negligible. We still have some nice buffer if I get four weeks.

You say: if everything goes well, our projection tells us that it looks like we should be able to ship by end of December.

Your customer hears: bla bla bla, bla bla, end of December.

They are thinking: Fred, our customer, has been asking for this feature for a while now, and their account has been shaky of late. I'll promise him the feature a few days before Christmas. The team will work harder (they always do, longer and on weekends) when I tell them how dire the situation is.

What actually happens: The customer checks in on the progress by the end of the first week. You just knocked off a 3-pointer faster than you expected. Your confidence is palpable and it shows. Customer is relieved, things are going better than plan.

But Monday morning welcomes you with bad news. Your datatable widget's filtering option isn't working ‐ a hitherto unknown bug. It seems fixed in a later version, but to use it, you'll have to upgrade your version of Angular, which simply can't be done now. There is a StackOverflow question with the exact problem you face, but it has zero answers. Your luck has run out, but you are no timid programmer. You roll your sleeves up and jump into the library code and fix it. But the delay has a cascading effect, and you are able to ship only by January mid.

That didn't go well. The feature was not a show-stopper for Fred, he was simply keeping us on our toes. However, we did promise him a timeline, and optimistic promises have a way of spreading, so he in-turn promised his VP and it became a big deal. He had to break his promise, which isn't boding well for your business.

The moral of the story is that no matter how many ifs, buts and other caveats we insert, the customer will treat our estimates with an inappropriately high level of certainty.

There are a few things we could have improved on though. At least next time, we'll:

Re-iterate that estimates are not commitments.
Ask what our estimates are going to be used for. We should always know the full implication of the numbers we provide.
Be careful to not amp up expectation in others because today's good weather does not translate into tomorrow's. But we respect our business owners and understand that they are capable of dealing with reality. We keep them aware of slips in the plan as they happen. We remain stoic in the face of both progress and setback.
Not work on weekends. We worked on weekends, and it didn't help. It never does, in the long run. Don't work on weekends, and don't give optimistic estimates thinking you would.
Relax and reflect. When the going gets tough, we forget to think. The stress creates adrenaline, and it clouds our judgement. The datatable filtering was not a show-stopper. It could have waited and we could have shipped on time, only if we had the presence of mind to consider the possibility.

Our estimates are going to come back and bite us ‐ that is the nature of the beast ‐ we ourselves become bound to it through anchoring and our customers become bound to it because they need a number to plan things around.

Casual estimates

Casual conversations are the minefield when it comes to dangerously wrong estimates:

"So, how hard is this feature? Is it like an hour's worth of work, or a few?"

And you say, "definitely not a quick one. A day, probably."

Stop right there and take note:

The customer did your job for you and supplied an estimate in the question itself. In fact they started from the lowest possible number, putting pressure on you to anchor on it. That is negotiation. Estimation should never be a negotiation.
We thought this was a casual question and gave a casual (aka wrong) answer. But when you get around to actually estimating this, telling the customer that the earlier estimate was wrong is akin to backtracking on a commitment. It wouldn't matter that it was a ballpark number, that detail will be forgotten by both the customer and you. Casual promises are promises nonetheless, so don't make them.
Have conversations based on time instead of story points. Story points are a useful abstraction, try to stick to it if you can. (Why use story points instead of hours for estimating).

Now the same conversation would go like this:

"So, how hard is this feature? Is it like an hour's worth of work, or a few?"

And you would take a moment to reflect on it, and say:

"I am not able to say off the top of my head. I'll estimate this story, probably need an hour to do that, and get back to you."

This is obvious but overlooked ‐ the accuracy of an estimate is proportional to the time & effort you spend on it. The more you invest, the better the estimate. And no matter how quick and approximate an estimate your customer needs, you cannot afford to be lazy and put out a number that you don't have maximum confidence in.

Estimation is a high-risk activity and should be treated with the gravity it deserves. Casual estimation underplays the risk involved and the time it takes to come up with a reliable estimate. It makes us rush through and produce unreliable numbers which ultimately leaves everyone worse off.

We should explicitly allocate time for estimation. This conveys the time & cognitive effort required to do it and ensures that stakeholders think through what they really need before they ask us to spend time on estimating them.

Now the question becomes what do we do with all that time we allocated for estimation? To answer which, I have to first hate on fuzziness.

Fuzziness

Fuzziness is a knot in the stomach. I feel it when I estimate a story without thinking it through. It is my body telling me that I'm painting myself into a corner. But fuzzy is easy; we see a story and put a number to it without expending the mental energy to articulate it.

We say this story is a 1-pointer or a 2-pointer based on a hazy intuitive notion. This is fuzzy, it doesn't work well when you have a short-term delivery horizon. We tend to underestimate the task at hand and overestimate our abilities, and no matter how many times we have been wrong, we fool ourselves that we can do better this time. We are also people pleasers, we hate giving bad news and tend to default on the most optimistic numbers.

Getting better at estimation involves replacing fuzzy estimates with concrete data. It is hard to defend estimates made based on gut feeling, but data can speak for itself. Data doesn't people-please ‐ it gives us the conviction needed to communicate well and set the right expectations with our customers.

We defeat fuzziness by concretizing, aka, planning a task to its actionable items.

Plan, don't estimate.

We have to know what we are estimating to be able to estimate it. Sounds obvious? It is not. We often don't know the specifics of a story when we estimate it; we simply do a high-level sizing and throw a relative number. We keep aside the figuring out of what needs to be done for the later implementation stage.

The reason we default to such fuzzy estimation has a lot to do with the economics of energy of the prefrontal cortex. This is the part of our brains that is responsible for decision making, prioritization and deliberate thought.

The prefrontal cortex is the biological seat of your conscious interactions with the world. It’s the part of your brain central to thinking things through, instead of being on “autopilot” as you go about your life.

The prefrontal cortex holds the contents of your mind at any one point. It’s where we hold thoughts that are not being generated from external sources or from the senses. We ourselves are generating them.

Conscious mental activities chew up metabolic resources, the fuel in your blood, significantly faster than automatic brain functions such as keeping your heart beating or your lungs breathing. The stage (prefrontal cortex) requires a lot of energy to function. It’s as if the lights are a long way back from the stage, so you need a lot of them, all on full, to see the actors. To make matters worse, the power to light the stage is a limited resource, decreasing as you use it, a bit like a set of batteries that constantly need recharging.

- Your Brain At Work, David Rock.

Deliberate thinking is a formidable productivity booster (duh!). But generating thought is energy intensive. So we resist it and rely on gut feeling, which is simply the brain cruising on auto-pilot. It is not an effective strategy. Here, look at what some upfront planning did to Rachel Aaron, a writer (emphasis hers):

Looking back, it was so simple I feel stupid for not thinking of it sooner. If you want to write faster, the first step is to know what you're writing before you write it. I'm not even talking about macro plot stuff, I mean working out the back and forth exchanges of an argument between characters, blocking out fights, writing up fast descriptions. Writing this stuff out in words you actually want other people to read, especially if you're making everything up as you go along, takes FOREVER. It's horribly inefficient and when you get yourself in a dead end, you end up trashing hundreds, sometimes thousands of words to get out. But jotting it down on a pad? Takes no time at all. If the scene you're sketching out starts to go the wrong way, you see it immediately, and all you have to do is cross out the parts that went sour and start again at the beginning. That's it. No words lost, no time wasted. It was god damn beautiful.

Every writing session after this realization, I dedicated five minutes (sometimes more, never less) and wrote out a quick description of what I was going to write. Sometimes it wasn't even a paragraph, just a list of this happens then this then this. This simple change, these five stupid minutes, boosted my wordcount enormously. I went from writing 2k a day to writing 5k a day within a week without increasing my 5 hour writing block. Some days I even finished early.

- How I Went From Writing 2,000 Words a Day to 10,000 Words a Day, Rachel Aaron.

She managed to increase her daily output from 2000 words to 10,000 words a day, mostly thanks to planning. Programming is also in its basic form, just writing. There are many lessons to be learnt from that craft, and this one is a case in point. Instead of wallowing in the code for hours ‐ going where the code takes us, ending up in blind alleys and backtracking, and getting lost in the trees and losing sight of the forest, a little bit of thought & planning can do wonders for our days as programmers.

The earlier we plan, the better. In fact, going into as much detail of a story as we possibly can before we estimate is the simplest way to increase the accuracy of our estimates. It will also bring out any hidden assumptions before we give out invalid estimates, or worse still, get neck deep into implementing the wrong thing.

Implementation becomes a breeze once we have a solid plan to follow. It is a beautiful day of writing code when that happens.

Estimate as a by-product of a plan

So what has planning got to do with estimation? It obviously makes for a more enjoyable programming experience. But it also brings our estimates to ninja precision. That is the whole point of this post.

Estimate is a by-product of a plan. It can't help but jump out at us when we look at a well thought out, written plan. At a sufficiently fine level of granularity, we would have the very implementation steps in the plan, and to come up with an accurate estimate, all we need to answer is how hard is it to convert this into code.

Drilling into this level of granularity might not be practical in many large agile environments. There will be too many people and stories for us to go into depth for each of them. But remember we are not talking about those environments. They have their own strategies which we cannot afford to have.

However, it is easy to make mistakes even in the relative sizing of stories as is done in a pure agile practice. Some of the stories turn out to be much larger than we anticipated. That is fine, the process accounts for such errors. But the feedback that the story was not estimated correctly is a lot more useful when we have a plan backing the estimate. Without a plan, the estimate is a one dimensional number, and that is all we are left with to evaluate any feedback on. But with a plan, the feedback is more useful ‐ it is easier to find out what went wrong in our assumptions and account for that when estimating similar stories in the future.

To conclude, plan early, plan often, plan deep. And good things will follow.

Getting started with AngularJS in Rails

2014-01-31T14:40:00Z

I have found that emails are how I get started with writing anything substantial that people find useful. The following post originated as an email to the London Ruby Usergroup when someone asked for pointers in integrating AngularJS with Rails. It was written in a hurry and is presented without any modification. Read on to listen to what I have to say about building an AngularJS project within Rails. I welcome your thoughts.

General

Use Coffeescript.
Start with Egghead.io. Read AngularJS by Brad Green, Shyam Seshadri. Spend a few hours and skim it cover-to-cover at least once. Angular documentation on the web is quite fragmented; a book is the best way to begin learning Angular.
If you aren't familiar with Underscore.js already, go through it. Add it to the project as soon as you begin. You'll need it.
Drop gem 'ng-rails-csrf' into your Gemfile to make Angular AJAX requests play well with Rails CSRF protection.

Directives

Some material on the web might lead you to think that a deep understanding of Directives is crucial to ‘get’ Angular. But for a beginner, the important thing is to just have fun doing things the Angular way (it is loads of fun, let me assure you). The basic directives Angular provides (eg: ng-repeat and ng-show) can go a surprisingly long way. Worry about transclusion, $compile, $digest and other arcane incantations only after you get comfortable with the basics.
One reason for you to write a directive is when you need to use existing widgets. A lot of popular widgets already have Angular wrappers. For example, we use the select2 wrapper for multi-select. There is also AngularStrap with a bunch of nice components that wraps over Twitter Bootstrap. The v1 (http://mgcrea.github.io/angular-strap/1.0/) version has Datepicker and Timepicker which isn't available in the new version yet.
You would hit a point when off-the-shelf directives won’t be enough and you'd want to do something that requires DOM manipulation. That would be a good time to start digging into Directives. You can also start exploring Directives when you start seeing patterns in your controllers that can be extracted out into reusable components.

Serialization

We use ActiveModel::Serializer. It is a safe choice.
Pay close attention to what you include in each serializer. Here is the obvious thing: you can put a bunch of has_many in your User model, and if you keep doing that for other objects, a render json: User.all will end up sending your entire db graph to the client.
We implement a base serializer for most entities with some basic fields and no associations. When we need more data/associations, we sub-class them for that specific use case. Discourse follows a similar style - https://github.com/discourse/discourse/tree/master/app/serializers
Try not to do any decoration on the server side; it is tempting with all the view helper goodness Rails provides, but then you are splitting logic between client and server and it causes all sorts of headaches. You can use Angular filters as client-side helpers on some occasions. For example, {{ start_time | date }} prints out nicely formatted dates.

Forms, validations and POJO form objects

Angular gives you validation attributes like ng-required, ng-minlength, ng-maxlength etc. on all form INPUTs. For simple use cases, this is the best way to go. But for a complex form, we ended up building an ActiveModel-like POJO Form Object to hold validation information, and for doing some slight transformations on the data before submission. We bound this object to the scope and removed a lot of clutter from our controllers that had begun to get bloated. We have to see how this pans out in the long run; I can give code snippets if you wish.
When I introduced Angular to our Rails project that was already under development, I tried to minimize the amount of change required by trying to keep form POSTs as they were.

<input type='text' ng-model='post.title' name='post[title]'>
<input type='submit'/>

It worked for the most part, but we ran into some edge cases that wasted a lot of time. That was when I learned to let go and fully embrace the Angular way. We switched over to something like this:

<input type='text' ng-model='post.title' name='post[title]'>
<button ng-click='submitForm()'>

And the controller:

angular.module('myApp').controller 'postCtrl', ['$scope', '$http', '$state', 'flash',
  ($scope, $http, $state, flash) ->
  $scope.submitForm = ->
    $http.post(
      Routes.posts_path,
      $scope.post           # this is the ng-model hash we bound the input boxes to.
    ).success((data, status) ->
      $state.go("posts.show", {post_id: data.post.id})
    ).error((data, status) ->
      if data['message']
        flash.error = data['message']
      else
        flash.error = "Could not create Post. #{data}"
    )
]
# `flash`  is from https://github.com/wmluke/angular-flash.
# $state.go is ui-router. https://github.com/angular-ui/ui-router. More on that later.
# we use JSRoutes to access Rails routes from JS. Quite useful. https://github.com/railsware/js-routes/

The above code isn't ideal. You will be better off putting the $http.post inside an Angular service. This looks like a good post describing that: http://sravi-kiran.blogspot.in/2013/03/MovingAjaxCallsToACustomServiceInAngularJS.html.

Rails+Angular

Go all the way in. Don’t try to keep a mix of Angular and non-Angular pages. Except for Devise, all our pages are rendered client-side. We don't have to worry about SEO since this is an internal app.
Use ui-router, not the one that comes with Angular. The ui-router README could be confusing, but like most of Angular documentaiton, it becomes obvious in hindsight if you persevere.
We keep everything together in the same Rails project. Here is a rough structure:

routes.rb:

root ‘base#angular’
resources :posts, only: [:index]

app/views/base/angular.html.erb:

<div ng-app='myApp'>
  <div ui-view></div>
</div>

posts_controller.rb:

class PostsController < ApplicationController
  respond_to :json

  def index
    render json: Post.all, root: false
  end
end

app/assets/javascripts/routes.coffee.erb:

angular.module('myApp').run ($rootScope, $state, $stateParams) ->
  $rootScope.$state = $state
  $rootScope.$stateParams = $stateParams

angular.module('myApp').config ($stateProvider, $urlRouterProvider) ->

  # The default route
  $urlRouterProvider.when("", "/posts/list")

  $stateProvider
    .state("posts",
      url: "/posts"
      controller: "postsRootCtrl"
      template: '<div ui-view></div>'
      abstract: true
    ).state("posts.index",
      url: "/list"
      controller: "postsIndexCtrl"
      templateUrl: "<%= asset_path('posts/index.html') %>"
      resolve:
        postsPromise: ($http) ->
          $http.get(Routes.posts_path())
    )

app/assets/javascripts/posts.js.coffee:

angular.module('myApp').controller 'postsCtrl', ['postsPromise', '$scope', '$http', (postsPromise, $scope, $http) ->
  $scope.posts = postsPromise.data

app/assets/templates/index.html

{% raw %}
<div ng-repeat="post in posts">
  <h1>
    {{post.title}}
  </h1>
  <p>
    {{post.body}}
  </p>
</div>
{% endraw %}

Quirks and bugs

It is worth keeping in mind that in JS {} == {} is false, since comparisons are by object identity, not by object value. You can do explict comparison using Angular.equals when you need it. But in some cases Angular does an implicit comparison, for which it doesn't use Angular.equals. For example, if you try loading a page with a bunch of pre-selected radio-buttons, if their values are JS Objects (Hash), Angular won't pre-select them since the comparison will fail. This is not an issue if your values are strings/numbers.
JS treats the keys of all objects as strings. a={1: "Hello"}; _.each(a, (value, key) -> console.log(typeof key)). If your server sends Hashes whose keys are numeric ids, they'll end up in JS with string ids.

A quick and dirty way to inspect an Angular model is to drop this in your Angular view template:

{% raw %}
<pre>
  {{ post | json }}
</pre>
{% endraw %}

There is also Batarang, a Chrome plugin which lets you inspect your scope interactively. But I've found it to cause my pages to misbehave in certain cases. I these days use angular-chrome-debug, which is quite light-weight. I load this script in my app in development env so that I don't have to go thru Chrome Snippets everytime.
Angular has Angular.copy for deep copy when you need it - https://github.com/angular/angular.js/blob/master/src/Angular.js#L725.

Well, that's mostly it. I am generally liking building rich UIs with Angular and wouldn't go back to building for the web without using data-binding.

Please keep in mind that the structure I described is working well for our app, which is not meant for public consumption. You might want to look at keeping your Angular and Rails project separately and utilize JS build tools like Bower, Yeoman and Grunt if your app needs that.

15 Things for a Ruby Beginner

2012-05-30T05:40:00Z

The following is a post I had recently sent the Bangalore Ruby User Group. It has been slightly modified to address a larger audience.

There were many Ruby beginners in last week's meetup, and the common question we heard was 'after the very basics, what next?'

The best way to learn Ruby best practices is to pair with an experienced dev; the way I learned was by inheriting a reasonably small, but well-written codebase from an amazing colleague. In the absence of either, here is a checklist of 15 things (since 'N things that you need to know about X' is the in-thing these days!) that I'd recommend a Ruby beginner to consider:

1. The very basics

Our very own rubymonk.com has a Ruby primer which was written for exactly this purpose; we open our inbox everyday to gushing feedback from people who've found it to be a great way to learn Ruby. Try it and let us know how it goes!

tryruby.org also has a basic introduction to Ruby, and has been around longer. Edgecase's Ruby Koan is an interesting concept, and covers the language both in breadth and depth, and is a very strong recommendation. It should take you anywhere between 5-10 hours to finish all of the Koans. Do try it!

I have heard good things about Learn Ruby the Hardway, but haven't tried it out myself. Okay, I just skimmed through portions of it and I'm not really happy - LRTH seems to be mostly a line-to-line translation of Python code to Ruby. It uses 'while' loop in places where equivalent Ruby idioms (Enumerables) would have made more sense. Also there is no mention of blocks, metaprogramming and duck-typing, which pretty much is a deal-breaker for me. But to be fair, the target audience for LRTH seem to be non-programmers for whom the concept of loops and objects would be new, and for them it does the job very well.

Wait, have you read Why's Poignant Guide to Ruby? If this is the first time you're hearing about why the lucky stiff, read this amazing piece on _why by the Smashing Magazine. And definitely read The Poignant Guide: http://mislav.uniqpath.com/ poignant-guide/. It is full of cats, foxes, chunky bacon, cartoons that doesn't always make much sense, space travel and what not. This was one of my first introductions to the Ruby community, and the guide lent the language and the community a fun, quirky and happy aura. You may or may not take away much Ruby knowledge from the guide - I couldn't when I read it for the first time. However you'll definitely understand some of the quirkiness and philosophies that influence the Ruby community. I am a huge fan of _why, and here is my favourite quote:

when you don't create things, you become defined by your tastes rather than ability. your tastes only narrow & exclude people. so create.

2. The ecosystem - RVM/rbenv, RubyGems, Bundler, Git and Github

I think all of these tools are mandatory for being a productive Ruby programmer. You'll encounter them soon enough:

RVM/rbenv: these are tools used to manage multiple Ruby versions on the same machine. I've been using RVM without complaint for quite a while, though there are people who will go up in arms against RVM for various reasons. As a beginner, you should be comfortable with either.
RubyGems: a gem for anything, a gem for everything. If you are using RVM, it will install RubyGems by default for you. http://docs.rubygems.org/read/chapter/4
Bundler: You'll learn it easy enough if you are using Rails. But even for non-Rails projects, Bundler is now the de-facto tool to handle gems and dependencies. It is one of those tools that when you see for the first time you would wonder how you ever lived without it.
Git: You are a git if you don't use git yet. If you are not even using any version control at all, good for you - there aren't bad practices that you need to unlearn. If you are on SVN, or God forbid CVS, jump now. http://git-scm.com/book/en/ Getting-Started-About-Version- Control
Github: You have a Github handle, right? 'nuff said.

3. Editor

I don't care. Pick one, use it well. If you're on Vim and is on Insert mode all the time, then use Notepad instead. It would be more productive. Learn your editor.

Here is a list of editors/IDEs people generally use for Ruby development:

- RubyMine
- Sublime Text
- Textmate
- Vim
- emacs

If you are using Sublime Text, install and use its corresponding Ruby package. Ditto for Textmate.

If you are on Vim, using the right set of plugins is a requisite to be productive. There is the popular https://github.com/carlhuda/janus and Srushti's https://github.com/srushti/vim-get which I use when I work with Vim. Even if you don't go for these plugin distributions, spend enough time to find the right plugins for Ruby development.

I don't know about the best plugins for emacs, but there are people who use emacs to develop in Ruby. Even Matz uses emacs; search and you shall find.

4. Ditch 'while', 'for' and array accumulation

Read this: http://martinfowler.com/bliki/ CollectionClosureMethod.html

An apparent sign of a programmer who does not know Ruby well enough is code that uses 'for' and 'while' loops for iteration and accumulation. I am hardpressed to remember occasions where I had to use them instead of the Enumerable methods #each, #map, #select, #inject, #reject and #detect. Learn these methods, chew on them, and use it everywhere!

(infinite loops are almost always written using the loop do..end construct though. but how often do you write infinite loops anyway?)

5. Hash

At the time when I started writing Ruby, the languages that I had written in for a reasonable period of time before were CA-Clipper, Borland Turbo C and some VB 6.

The first two did not have a hash, associative array or dictionary - whichever you prefer to call it. As to VB6, the only thing I can remember is DataGrid and ADODB. Ah, the failed promises of drag and drop programming!

So Hash was a revelation and I started using it anywhere and everywhere. Do you want to build a CRUD app to manage customer info? Forget databases, I'll build a Hash and serialize to and deserialize from a YAML file. There were even more crimes committed using Hash that I dare not mention here. You would have gone through enough exercises that uses Hash when working through RubyMonk or Ruby Koans. But if you haven't, make sure you understand Hashes well enough. Specifically:

iterating over a hash
assigning default values for undefined keys in a hash
Hash#keys and Hash#values for extracting just the keys and values
In Ruby 1.8 Hashes are un-ordered: ie, you can't rely on the ordering of the hash to be same as the order in which you added elements. In Ruby 1.9, a Hash is sorted on the basis of order of insertion.

6. JSON and YAML

These are not Ruby specific concepts, but find great use in the ecosystem. Know them well, they'll come in handy.

7. Understand Immutability and how Ruby passes object references around

This has slightly got to do with the above point - all the Enumerable methods are immutable, and it is a good introduction to how functional Ruby veer towards immutability. Immutability is more of a good programming practice than a Ruby specific idea - it helps you write clean predictable code, leave aside concurrent programming and race conditions. A method that mutates its parameter is a vile creature, don't bring it forth into existence.

If you come from a C programming background, building new objects willy-nilly would be a little hard to digest. So much memory put to waste! I remember reading somewhere that programmers who use high level languages leave a higher carbon footprint because their code is inefficient. I leave you to ponder over it.

For understanding some quirks around Ruby's immutability and interesting effects of how Ruby passes object references around, figure out where Array#clone is used. I remember wasting many a debugger breakpoint during my early days of Ruby because I didn't realize this. Don't let that happen to you! Understand the difference between a shallow clone and a deep clone. Even better, go write your own deep_clone routine! (limit yourselves to objects that can have strings, numbers, arrays and hashes)

Also read: http://ruby-doc.org/docs/Newcomers/ruby.html#objects

8. Ruby's object hierarchy

# All objects are instances of the class Object.
"a string".is_a? BasicObject # true

# All classes are instances of the class Class.
String.is_a? Class # true

# Class is a subclass of BasicObject.
Class.is_a? BasicObject # true

# Class is not an instance of BasicObject
Class.instance_of? BasicObject # false

# BasicObject is an instance and a sub-class of Class
BasicObject.is_a? Class # true
BasicObject.instance_of? Class # true

Okay, I lost it. It is pretty crazy: http://stackoverflow.com/questions/4967556/ruby-craziness-class-vs-object. As a beginner, you wouldn't need to understand the nitty-gritties. I've been programming in Ruby for about three years now, and it still confuses the heck out of me.

For now it is safe to understand that BasicObject is usually the root object of all objects in Ruby. And everything in Ruby is an object. This has a very useful side-effect (try this in IRB):

"some random string".methods - Object.methods

Also,

Array.new.methods - Object.methods

The above commands will show you methods that are specific to just strings and arrays, excluding all methods that are always present in any Ruby object (inherited from Object - like instance_of, is_a? etc.).

You might have noticed that the '-' operator gives you the difference between two arrays. Whenever you need a general purpose method and wonder whether Ruby comes with it, just try some plausible syntax in IRB. You might be surprised at what you find.

Even though Ruby lets you Object Oriented and procedural code, the language leans toward OO. Ruby treats even methods as objects:

"some string".method(:length) # gives you an object of the Method class.

The method object can be asked to run by invoking the 'call' method on it.

9. Creating your own Objects

Did you notice that the title wasn't 'Creating Classes'. That was one of the most useful advices I've ever received: Always think in terms of objects - not classes. Thinking in terms of Classes can subtly make you evolve your design upfront. Don't. Let your objects guide you in how your class definition should look. As a rough analogy, when building a home, the blueprint is valuable only as a reference for building the actual home. You imagine what your home should look like and draw a blueprint accordingly, not the other way round.

Start with sparse classes, add methods and attributes as your objects demand it. Srushti puts it better: Imagine you're an instance, and think about what you want to do and how you want to do it. You don't want to give up your secrets (encapsulation). You don't ask other people for information so you can do their work for them, you just tell them to do stuff for you (tell, don't ask)

Ruby has a very simple syntax for defining classes and building objects. If you come from a Java/C# background, it'd be the first thing you look for. But even if you are a die-hard procedural ninja, trust me, thinking in terms of objects will help you write better programs, tackle complexity and be a more capable programmer.

So, what are the things that are specific to Ruby that you need to be aware of?

- Message Passing. "abcd".length is in fact "abcd".send(:length).
- Module vs Classes (hint: they're very similar, but you can't instantiate a Module)
- Mixins (Ruby's answer to multiple inheritance and the greatest thing _before_ sliced bread)
- attr_reader, attr_writer, attr_accessor.
- instance methods and class methods
- instance variables and class variables.

And we all know that you don't use class variables unless you have a very good reason. Class methods aren't that bad, but are usually a smell. Whenever you find yourselves writing a class method, take a step back and make sure it can't be rephrased as an instance method, perhaps in a child object?

There is a lot more to OO, some less specific to Ruby. As you go deep into the rabbit hole, ponder over these blanket statements:

- Primitives (Hash, Array etc.) are evil! Build objects.
- Inheritance is evil! Use Composition.
- Conditions (if..else, switch..case) are evil! Use Polymorphism.

10. Ruby is interpreted. It is malleable. Use that to your advantage.

Interpreted programs are almost always slower than native code (which includes JIT). By choosing to use such a language, you are accepting a compromise in the speed/efficiency of your programs. But this gives you a great advantage: the flexibility to change your code at runtime. Though we can't claim 'code is data, data is code' like those hipster LISPers do, there is tremendous power in the dynamism (no reference to type systems) of Ruby. Learn it, use it, change the world!

I had briefly mentioned the 'send' method that is available for every object in Ruby:

"abcd".send(:length)

is same as

"abcd".length

That means you can do things like this:

puts "Hi, which method do you like to invoke on a string today?"
method_name = gets.strip
puts "a random string".send(method_name)

Did you see that? Unlike fully compiled languages like C/C++, Ruby lets you call arbitrary methods during runtime! (you can pass arguments to the method as parameters to the 'send' method)

Leave aside calling arbitrary methods, running arbitrary code during runtime is a breeze:

puts "What code do you want to run today, dear sir?"
arbitrary_code = STDIN.read # press ctrl+d to stop input
eval(arbitrary_code)

Try it, type in some short valid Ruby code and see it in action.

Now that you know 'eval' exists, forget about it. It is too dangerous to be almost ever used. It is unsafe and unscoped, but there are better things to achieve similar and useful results. The point of this exercise though was to see Ruby's dynamic nature in action. Since Ruby is interpreted, there is no limitation on what can be done during runtime. This can be used to great good as we will see in Metaprogramming.

11. Metaprogramming

Metaprogramming in Ruby more or less gives you ways to create/remove/redefine methods at runtime. If you have used Rails, you would have seen that you would write something like

class User < ActiveRecord::Base
end

and magically, the User class gives you methods like user.name, user.find_by_name, user.find_by_name_and_id. Depending on the fields in the database, Rails defines methods for you to use. This uses Metaprogramming where Rails defines the methods at runtime after consulting the table schema.

(talking about 'magic', usually when someone complain about 'magic' in Ruby code, she is most probably referring to some sort of metaprogramming in the code)

Metaprogramming is one of Ruby's most powerful concepts (anything borrowed from FP is yummy!), but it is open to use and abuse. They say that someone who knows metaprogramming well enough, but not enough to know where not to use it, is a danger to himself and society. The internet is rife with discussions around it and you'll find no shortage of flame wars, opinions and thankfully, documentation.

These are the methods you would want to look up to get a decent overview of metaprogramming in Ruby:

- define_method
- method_missing
- instance_eval
- class_eval

I would also recommend Yehuda Katz's excellent explanation of Metaprogramming by relating it to the context of 'self': http://yehudakatz.com/2009/11/ 15/metaprogramming-in-ruby-its-all-about-the-self/

Metaprogramming is a bit advanced, and if you don't understand all or any of it the first time, don't worry. Come back and take a look again later. Rinse and repeat. It is an acquired taste, give it time!

12. Closures (Blocks, Lambdas et al.)

Blocks are my favourite. They move mountains. Rather, they let you write beautiful DSLs when coupled with the right dose of metaprogramming. Have you seen factory_girl's syntax? It is an unholy mix of method_missing and 'yield'.

Factory :user, aliases: [:author, :commenter] do
  first_name "John"
  last_name "Doe"
  date_of_birth { 18.years.ago }
end

It is not really hard to build a DSL that reads like this, and there is no better resource to learn all of this than http://rubysource.com/ functional-programming- techniques-with-ruby-part-ii/. The first part of that series looks into the functional and immutable aspects of Ruby, and is also a recommended read: http://rubysource.com/ functional-programming- techniques-with-ruby-part-i/

13. Styleguides

Whenever you are in doubt, or the self becomes too much with you, go read the Ruby style guides.

Github's simpler style guide: https://github.com/styleguide/ ruby

The comprehensive one: https://github.com/bbatsov/ ruby-style-guide

14. Simplicity is virtue

Knowing what constructs to use where is a matter of knowledge and experience. Every approach has trade-offs in terms of readability, maintainability and efficiency. The battle between these have been the recurring theme in the battles programmers fight in their heads for years. Knowing the the trade-offs will help you make more informed decisions, but it might not always be enough. Some things need to be tried, tested and failed, and that is fine.

But be vary of Premature Optimization. When you have a choice between clever, short and maybe faster code Vs longer but readable code, go for readability. Ruby makes it easy to write really bad code that people would fear to touch with a long pole. It also lets you write beautiful and concise code. When you contemplate between the two, remember the joke about the psychopath who'll inherit your codebase, knows where you live, and pings you from your local network! The choice is yours.

15. None of this matters

If you are overwhelmed by this document or any links referenced from here, just ignore it. Remember the 10,000 hours rule. Happily go about writing code the way you know best! And write a bit more code. Try to pair with someone who knows things a bit more. Go read some well-written Ruby code from Github. Then come back and see what you've learned.

None of this is rocket science, but it takes time and practice for concepts to sink in, and that is just fine.

Have further questions? There are tons of resource on the internet to answer your questions!

Join one of your local Ruby Usergroups. The Ruby community is extremely helpful and accomodating towards newbies. Check this page to locate a usergroup near you: How To Find Ruby User Groups

Participate in the usergroups, ask your questions. Also hop on to Ruby's IRC channel #ruby-lang on Freenode. Irrespective of the forum, just make sure that you give enough context about your question to help others understand your problem. If you haven't read ESR's "Ask questions the smart way" yet, *this* is the time. Go read it now and get enlightened on the ways of the interwebs!

And remember to have fun! In Matz's own words:

For me the purpose of life is partly to have joy. Programmers often feel joy when they can concentrate on the creative side of programming, so Ruby is designed to make programmers happy.

Happy hacking!

B-Trees, large volumes of data and branching factor

2010-10-26T12:15:00Z

B-trees can be slow

I've noticed that after a few million keys, almost all b-tree based databases starts to get slower. This applies to both Relational/ACID-compliant as well as the NoSQL DBs. I tried Harbour's BTree implementation, TokyoCabinet, SQLite and MySQL. All of them started to slow down at various stages and the maximum I could go without an unacceptable decrease in performance was close to 40 million keys with TokyoCabinet in my 2GB RAM laptop.

Large branching factor = better performance

A binary tree is a tree which has two children per node, and has a time complexity of O (log₂ N) for any search operation. B-tree is a variation of binary tree in which there are M number of children per node. It thus achieves a time complexity of O (log_M N) for each search operation. M is called fanout or Branching factor. A large branching factor is the reason why the b-tree is a fast data structure.

When the number of keys (N) crosses a certain bound, the height of the tree will increase and every search/insert might need to traverse one more level. If this level is not in memory, the operation will need a disk access. Since disk access is expensive, this leads to bad performance. Thus,

Efficiency of a B-tree ∝ Available memory / F(N).

F(N) is a function on the number of keys and gives the height of the b-tree.

This leads us to the branching factor - given enough memory, increasing the branching factor will make each level of the tree hold more nodes and thus reduce the b-tree's height. Fewer levels means fewer disk access (if any) and better performance.

Disproportionally large branching factor = decreased performance

In an ideal world, to cope with larger values of N, we could simply increase the branching factor and individual node capacity to maintain the performance of the B-tree.

But the b-tree structure becomes a burden if the branching factor is so big that the first few levels cannot be completely held in memory. At this stage, accessing each level can trigger a page-fault causing disk swapping and very poor performance.

Two major concerns here are :

1. Limited Memory: The b-tree parameters are limited by the available memory. Since all search starts from the top, we need the first level of nodes to be completely in memory. More the number of nodes thats fits in main memory, the lesser the need for disk seeks. But scaling physical memory is an expensive proposition and infeasible beyond a point.

Solution: Sharding

Beyond splitting db load, there is a huge advantage with sharding - sharding a b-tree storage enables us to limit the number of keys that a single b-tree need to hold. Thus each b-tree will produce optimal performance.

Sharding the database along with its b-tree indices into different physical machine is one of the most effective scaling solutions for growing data volumes.

MongoDB uses this approach, and seems to be scaling very well judging from a few instances of MongoDB in production use.

2. Databases need constant tuning: Changing the database system parameters (branching factor, shards, indexes) etc. usually require downtime. Having scaling related down-times is a major pain and we are still yet to see a db that tunes itself to increasing data volumes.

Solution: Cache-oblivious structures and algorithms

The implementation of most algorithms that deal with large volumes of data makes advantage of the caching characteristics of the target machine. For example in B-trees, the branching factor should be such that a complete level of node should completely fit in the main memory. Cache-oblivious algorithms remove the dependency on the cache characteristics of the platform and implement data structures like b-tree to perform well in diverse memory architectures.

Prof. Charles E. Leiserson of MIT heads the Supercomputing technologies group which is doing pioneering research into cache-oblivious algorithms. There are quite a few papers on this topic at the MIT SuperTech website.

The primary authors of the cache-oblivious B-tree papers at CSAIL, Michael A. Bender, Martin Farach-Colton and Bradley Kuszmaul have founded Tokutek, which built a product based on cache-oblivious b-trees - TokuDB an add-on database engine to MySQL that implements fractal tree which is claimed to perform orders better than either InnoDB/MyISAM at high data volumes. Though shipped with MySQL, the implementation is not open-source.

Conclusion

B-trees are one of the most used data-structures in computing. However they are not a silver-bullet for every kind of data storage and retrieval problem. Developers still need to go down a few levels of abstraction and tune the database to gain the most from the DB. Sharding is a viable solution but it requires the application to be tailored to shard the dataset according to properly identified key fields. Cache-oblivious b-trees are yet to become widely available in mainstream databases.

It has been about 30 years since the first commercial RDBMS (Ingres) by Michael Stonebraker became popular, and the relational model of database with b-tree as its preferred storage structure chugged along. However in recent years, the demands of realtime web-scale applications have paved way for more research and changes in the database landscape.