Broken Rails 2.3.3 Tests

Fixtures are loaded differently in Rails 2.3.3+. If you are using FooBar.first instead of foo_bars(:one) in your tests, you may experience breakage. Hope this helps!

–DT

It’s All Middleware

Jon Crosby has an excellent talk from Mountain West Ruby Conference 2009 about how middleware is taking over web development. His article “A World of Middleware” is a bit difficult to parse if you don’t know Rack, so I’ll try to simplify what he’s saying a bit…

Rails 2.3 is built on Rack, which is a simple specification to connect web frameworks like Rails and Merb with web servers like Mongrel and Thin.  Rails 2.3 also lets you define middleware code to be run before Rails is invoked, again using Rack.  In Rails terms, middlewares are like before_filter’s that are run before Rails is ever even called.  Another way of looking at a middleware is like a miniature application that can connect, via Rack, to another middleware.  Each middleware stack handles certain requests (for example, certain url’s can be handled by specific middlewares), or all requests.  If the middleware returns 404, the next middleware is called.  Rails 2.3 only handles requests that fall to the bottom of the middleware stack.

Crosby’s point is, why bother with a chain of middlewares leading up to an application?  All you need is a chain of middlewares that are the application.  His example in the talk is an authentication middleware that sits on top of everything else, instead of the authentication layer being part of a monolithic application.

The best part is that middlewares can be created as black boxes, and simply included in your Rack application (or used as a middleware in your Rails 2.3 application).  And it looks like the best tool for creating simple middlewares is Sinatra.  This stuff isn’t even on the main Sinatra site yet, but you can read about it on the Sinatra blog.  To quote: “multiple Sinatra applications can now be run in isolation and co-exist peacefully with other Rack based frameworks.”  This is awesome – it means that a Sinatra application can be used as middleware for a Rails application (or other Sinatra applications).  So it’s easy to build a big application by chaining together smaller Sinatra middlewares.

Since Ruby’s open source community is so strong, there’s no doubt we will be seeing pluggable middlewares that fit easily inside any Rack application.  This is exciting stuff, and I look forward to building an app this way!

Clarifications to My CouchDB Post

I got great feedback on my previous post about CouchDB, and I wanted to clarify a few points.

mattly pointed out that there is still need for a middleware tier, since client-side Javascript is not a reliable way to do validation or security, since Javascript can simply be turned off.  Andrea Schiavini wrote that Rails is much more than a database tier, and that Rails has plenty to offer beyond ActiveRecord (Rails’ database connector). Both objected to my referring to middleware tiers like Rails as “obsolete”.

Regarding mattly’s point, I really should have said “most of this is going to be obsolete”, not all.  He gives several examples where a middleware tier is necessary because CouchDB doesn’t have a system in place to deal with certain tasks (like comment spam detection).  However, my understanding is that CouchDB’s upcoming security model really is supposed to replace the middleware tier.  What prompts me to say this is the focus on “CouchApps”, which would run on the user’s computer (CouchDB and all), and synchronize the database with a master (“Eventual Consistency”) when the client was finished.  In essence, it’s a way to cache the database on each client.  Security/validation then becomes a matter of which client databases are allowed to update the master.  As far as I understand, it’s very similar to the git source control security model.  More here.

For a more traditional web-based application, I would put CouchDB behind a firewall, open up GET’s via a proxy (like Apache) but restrict other methods to go through a very simple middleware tier (this helps with Javascript domain restrictions as well).  The middleware tier is not completely gone, as mattly correctly noted, but it’s down to a handful of scripts, not a full-blown application stack.  This assumes that CouchDB’s upcoming security system doesn’t already provide this functionality.

As far as Rails goes, I happen to be a full-time Rails developer, so I’m quite familiar with the benefits of Rails!  However, much of what Rails offers beyond ActiveRecord is also deprecated in CouchDB’s model.  For example, routing becomes less important when you’re building an AJAX-driven application with CouchDB.  Simply grab the view or document you need with AJAX, and render your web page client-side based on the result.  CouchDB takes care of the routes to the various views and documents already, so you’re left with a few static HTML documents and Javascript libraries that don’t particularly need routing.  Environments in Rails are also less useful when working with CouchDB.  In CouchDB, each database is its own environment.  As long as the databases have the same name on each server, no code has to change.

Anyway, thanks for the feedback, and I look forward to talking more about CouchDB in the future!

:-Daniel Tsadok

Database Disruption From The Couch

Web development typically follows a specific set of steps. There are different workflows for different people, and I’m not trying to establish a best-practices workflow here (just the opposite, as you’ll see), but the differences are usually in the order in which the steps are performed, not the actual steps themselves.

The first step (again, conceptually the first, not necessarily chronologically) is to design a database for the application’s needs – the database tier. That means breaking down the various components of your blog into tables. For example, if you are building a blog engine, you would need at least a table for articles and another for comments.

The second step is to write a “middleware” tier – that just means software that sits in between the client and the the database. You don’t want people to connect to the database directly or they’d be able to wreak havoc. The middleware tends to be where you do things like keeping track of who’s allowed on what page.

The last step would be to design a “view” or “presentation” tier to let your clients interact with your database in a (hopefully) safe manner. The view tier, among other things, takes the data that is broken up into tables and puts it back together again.

Ruby on Rails, which I’ve loved for years, excels at connecting these dots. ActiveRecord, which I consider the crown jewel of Rails, does a fantastic job at simplifying the complex job of creating and managing the database, for example. Rails also handles the middleware tier quite well with ActionController. The view tier (ActionView) doesn’t add all that much, but server-side view tiers rarely do when it comes to the web.

This is all going to be obsolete with CouchDB*.

What makes CouchDB so damn disruptive is not the fact that it beats frameworks like Rails – it’s that it completely side-steps them.  CouchDB dispenses with structured tables altogether, relying on “documents” instead. Documents are arbitrarily complex data structures that contain as much or as little data as you need. The (partially completed) free book on the official site uses the example of the business card.  Some business cards have fax numbers, and some don’t.  Some have website addresses, and some don’t.  In a classic, structured database, you would need to have a column in the database for fax numbers, even if only one business card in your database has one, because the structure of the database has to be in place before the data is added.  If the structure doesn’t accommodate the data, then the structure has to be altered, which is always a potentially dangerous thing (Rails handles this with migrations).  CouchDB lets you put whatever data you want in whatever document (in this case, business card) you want.  That means that instead of having to add a fax number column to every business card in the database, you could simply add a fax number to the new document and only the new document.

ActiveRecord is amazing at simplifying all the structured SQL stuff.  But once that complexity starts to go away, the value added by ActiveRecord starts dropping off dramatically.  If the database tier is already super-simple, why bother with a fairly hefty helper library on top of that?  CouchDB itself makes ActiveRecord (and other fancy ORM’s, like Hibernate) largely obsolete (and even quaint).

Another cool feature of CouchDB is that it uses HTTP for all interactions.  That means that the database runs on a web server.  Since CouchDB has built-in HTTP features like authentication, cookies, and caching, there’s not much the middleware tier is needed for, especially given that CouchDB works directly with AJAX, so much more logic can be put in the browser.

The last thing is the view tier – getting the data back to the user**.  CouchDB uses something called MapReduce, invented at Google, to deal with searching through existing data.  It’s a little tricky, and possibly the only thing preventing me from jumping fully on board with CouchDB.  Instead of SQL queries, you write “view functions” for CouchDB.  The end result is pretty much the same.

As an aside, one of the coolest thing about view functions is that they can be stored in CouchDB directly.  You just give them a unique identifier (like “view-blog-articles-by-date”) and store them as documents in CouchDB.  That means application code can be stored in the database, which is great because backing up the database will backup a good chunk of your code as well.

So in conclusion, a lot of what makes Rails great, and it is great, is deprecated by this humble open source application.  I didn’t even get to the built-in scalability that CouchDB has going for it.  CouchDB is a Big Deal, and I’m keeping my eye on it.

–Daniel Tsadok

* To qualify slightly, the concepts introduced by CouchDB are going to be disruptive, whatever form they take (i.e. even if CouchDB itself doesn’t take off).

** I’m fudging the presentation tier and the view tier a bit, in case you’re nitpicky ;-)  This post is not MVC-compliant.

My jQuery Tutorial

As part of my school’s “DriveBy” program, where students teach other students about various topics, I wrote a tutorial for the awesome jQuery library for Javascript. Check it out!

The Rise (and Approaching Fall) of the Duopoly

I wish I had a nickel for every time I heard someone say they couldn’t “get on the internet”.  Whether it’s via cable, DSL, 3G, or whatever, people are always having problems logging on.  It’s a baffling problem for me, because the internet evolved from ARPAnet, which was commissioned by the US Department of Defense forty years ago to be a completely reliable network.  So why, decades later, is “the internet” so “unreliable”?  The answer is that it’s not.  Individual connections (like mine and yours) may not be reliable, but the internet as a whole is incredibly reliable.  Any device connected to the internet can connect to billions of computers all over the world, seamlessly.

Ok, so why is your connection so unreliable? Don’t be so hasty to blame your service provider, although they certainly deserve it.  The real blame lies collectively with every consumer who currently pays for internet access.

One of the things that saddens me about the evolution of the internet is how it went from being a participatory network to a provider-consumer one.  Originally, the internet was a community of computers. Later, but before the broadband revolution, there were dozens of companies that offered internet via dial-up, which has a maximum speed of 56K, paltry by today’s standards. Broadband made dial-up obsolete, and the dial-up business crumbled.  Unfortunately, broadband required a direct connection to the home, and the only companies that could make that connection were the phone and cable companies.  Startup broadband just didn’t work.

Which brings us today, where your only options for wired broadband in most of New York City are Time Warner and Verizon (there are some small DSL companies that still exist out there, but they usually require you to have Verizon phone service already).  In other words, New York City residents face a duopoly when it comes to broadband internet service – they must choose between two mediocre, unreliable, bloated bureaucracies.

Considering how long the technology behind the internet has been around, surely there are other possibilities?  I believe there are, and in the coming years, I believe those possibilities will crystalize.

More on that in another post…

–DT

Time To Phase Out Rails Constants

I haven’t seen this documented anywhere, but it looks like Ruby on Rails picked up something nice from the Merb playbook: The Rails class now has the properties root and env. That means that instead of typing RAILS_ROOT, you can type Rails.root (just like Merb.root). Why bother, you ask? Because it’s much cleaner to keep all the Rails configuration stuff in one namespace. It also looks nicer in code.  Mainly, it’s something I liked about Merb that I thought was missing in Rails, so I’m happy to see this kind of positive give-and-take between the two communities. RAILS_ROOT and RAILS_ENV are probably not going anywhere anytime soon, but I think this a step in the right direction.

Nature And The City

I saw The Waterfalls in New York City on Labor Day, and at first I didn’t like them.  For one thing, I had the chance to visit Niagara Falls relatively recently, and it’s hard to get excited about these relatively puny ones.  For another, The Waterfalls looked artificial, which of course they are, but almost intentionally so.

But The Waterfalls provoke questions about cities, humankind’s primary assertions over nature.  The city is perhaps the most artificial thing in the world, but it is still built on a foundation of nature.  And nature comes back into the city in subtle and sometimes spectacular ways.  The spectacular ones are well covered by the evening news, so I’d like to focus on the subtle.  Go to any tree in the city, and look at the sidewalk around it.  Chances are you will be able to see cracks in the sidewalk leading towards the tree, caused by its roots.

But wasn’t the tree planted there by the city’s parks department?  Absolutely.  It is something natural on top of something artificial on top of something natural.  There may well be more layers that I don’t know about.  But the tree was artificially placed there, out of context, in an artificial environment.  And of course, the tree does not conform – hence the cracks in the sidewalk.

Which brings me to The Waterfalls, which are artificial imitations of a natural phenomenon.  Perhaps they look as artificial as they do to remind us what a city is all about – how natural and engineered will always have an uneasy coexistence, and how our attempts to superimpose nature onto an artificial landscape will always have unintended consequences.

It’s impossible to make something connected to the Brooklyn Bridge look natural.  So why try?

Rest In Peace, Randy Pausch

Randy Pausch, the professor famous for his “Last Lecture”, which inspired so many people, including myself, has passed away at the age of 47.  My thoughts are with his family at this difficult time.

(The link I provided is down, for now.  Here is a video of Professor Pausch’s famous lecture).

Talking About TED

I’ve been really into TED lately – it’s an annual conference featuring a series of talks by a variety of thinkers.  There are all sorts of topics covered, like “The amazing intelligence of crows”, by Joshua Klein, or “Why we know less than ever about the world” by Alisa Miller, or “A journey to the center of your mind” by Vilayanur Ramachandran.  It’s really amazing stuff.  I was thinking of making a DVD of my favorite talks so I could share them with others, which I can do legally because the TED talks are both downloadable and shareable under the Creative Commons license.  In terms of “bandwidth” (i.e. information per second), it’s one of the best resources out there to expand your mind.