BASH howto – find files with required text in an svn managed folder

This is just a bash (linux/mac terminal) one-liner to look for files with matching text in my rails app, which i do all the time.  It’s a lot faster and more useful than Nautilus’ file search, and is useful when you don’t use any IDEs.  Personally i do everything in the command line and gedit, nice and simple.

Here i’m looking for the string “ratings” in my views folder and my public/javascripts folder

This is broken down as follows:

get all files in these folders
| surround them with quotes (otherwise filenames with spaces would screw up the later steps)
| only consider non-svn files (i’m not interested in the files svn uses to manage things)
| look inside the files for the search string

And here it is:

find app/views public/javascripts -print | awk ‘{print “\”"$0″\”"}’ | grep -v \.svn | xargs grep -i “ratings”

Obviously this is a bit of a mouthful but once it’s been done it will go into your command memory, and you can get it back by pressing ctrl&r, and starting to type ‘find’.  You may need to press ctrl&r a few more times to get the right command from memory.

Note – wordpress or your browser may change some of the above double quotes to weird web quote characters.  I really fracking hate that.  Anyway, when you paste it over, change all the quotes to normal single or double quotes.

Distinguish between local production mode and server production mode in Rails

I just ran into a problem with production mode. Normally i never run this locally but i decided to use it to run my selenium test battery against, since it’s much faster than dev mode. In production mode on our server we pull our assets in from an amazon asset server, done with this line in production.rb

config.action_controller.asset_host = “http://assets.fakedomain.ir.com”

In my local version of production i don’t want to do this though, i just want to use what’s in my local public folder. I decided to have a look at ENV which i did by making apache explode prior to the above line –

raise “ENV = #{ENV.inspect}”

Quick restart of my local apache, reload the page, and BANG, i can see this:

ENV = {“APACHE_PID_FILE”=>”/var/run/apache2.pid”, “PATH”=>”/usr/local/bin:/usr/bin:/bin”, “LANG”=>”C”, “APACHE_RUN_GROUP”=>”www-data”, “APACHE_RUN_USER”=>”www-data”, “PWD”=>”/etc/apache2/sites-available”, “RAILS_ENV”=>”production”, “HOME”=>”/home/max”, “RAILS_ASSET_ID”=>”"}

aha – “HOME”=>”/home/max” – perfect!

So, i changed the line (and removed the raise!) to

config.action_controller.asset_host = “http://assets.fakedomain.ir.charanga.com” unless ENV['HOME'] == “/home/max”

Bingo. Perhaps not the cleverest way of doing it, but it’s safe – for everyone else, including our server, ENV['HOME'] will be something else, or perhaps nil. Either way, they’ll use the remote assets site.

Stubbing methods (eg callbacks) in a Machinist blueprint

I had a problem recently where i was using the test object-generating plugin Machinist to make a blueprint for one of my model classes. The model has a before_create filter which references some other data in the database, which didn’t exist in my test version of the db. This before_create filter had nothing to do with the tests i was running and i wanted to just stub it out. I couldn’t work out how, so wrote a wrapper method that looked like this:

def make_music_service(attributes = {})
  music_service = MusicService.make_unsaved(attributes)
  music_service.stub!(:set_default_column_customisations).and_return(true)
  music_service.save
  music_service
end

That meant that whenever i wanted to make a MusicService in my tests i had to call this method instead.  Pretty clumsy, eh.  But – Machinist creator Pete Yandell came through with a great bit of info on the machinist mailing list – when you’re inside your blueprint, you can reference the object being made with ‘object’.  So, now i can put the stub inside the blueprint, like so:

MusicService.blueprint do
  name     { Faker::Name.name }
  address1 { Faker::Address.street_address }
  address2 { Faker::Address.street_address }
  address3 { Faker::Address.city }
  postcode { Faker::Address.uk_postcode }
  default_password { "password" }
  object.stub!(:set_default_column_customisations).and_return(true)
end

Thanks Pete!

Firebug & jQuery – making other people’s sites better

Was just on a website that had 80 checkboxes (all checked) for various options. I wanted to have only one checked. There’s no ‘Uncheck all’ button. Design Fail! But – open the firebug console and run

jQuery(“input[type='checkbox']“).attr(‘checked’,false)

Bing, all unchecked. I get a childlike sense of wonder and joy from that, still. If only more things in life could be improved with a little console activity.

jQuery(“#pub.crap”).attr(“ale-count”, 20)

Gotcha with hand-rolled sql in rails associations – eg self.id and has_many

Sometimes in rails associations (eg has_many) you can’t quite do what you want with the existing options and have to do a bit of hand-rolled sql. And, often you want to refer to the object that the association will be called on. BUT – the following doesn’t work. (i’ve used a trivial example here, we wouldn’t normally hand-roll in this case)

has_many :foos, :finder_sql => "select * from foos where bar_id = #{self.id}"

The reason this doesn’t work is that the string has double quotes, which means that #{self.id} will be evaluated when the string is evaluated, which, because this is a class method, is at class load time! That means that self.id will come out as the id of the class – not what we want.

To get around this, use single quotes instead:

has_many :foos, :finder_sql => 'select * from foos where bar_id = #{self.id}'

With single quotes, the #{} block isn’t evaluated and sits there, waiting for the association to be called. When that happens, the finder_sql is combined, and the eval block evaluated, and at that point self IS the current instance, and the association will work. Bear in mind that, because we’re now using single quotes to wrap the string, any single quotes you might have in your sql will now need to be escaped:

has_many :foos, :finder_sql => 'select * from foos where flavour = \'chicken\' and bar_id =#{self.id}'

The shocking truth about rails callbacks! (mind answering my poll?)

Ok, the title’s a bit overly dramatic.  But i recently stumbled upon a feature of rails callbacks (methods called by before_save, after_create, etc) which meant my code was breaking, in a silent way.  After speaking to a few people about it, it seems a lot of people were unaware of this as well.  If you don’t want to read me waffling about my voyage of discovery then would you mind just skipping down to my poll about this?   I’m trying to get a sense of how many other people didn’t know about this either.

Halfway down the api page documenting callbacks ( http://www.railsbrain.com/api/rails-2.2.2/doc/index.html?a=C00000640&name=Callbacks ) , we read this:

Canceling callbacks

If a before_* callback returns false, all the later callbacks and the associated action are cancelled. If an after_*false, all the later callbacks are cancelled. Callbacks are generally run in the order they are defined, with the exception of callbacks defined as methods on the model, which are called last.

Now, i noticed this only because i encountered what seemed to be a bug in my code.  A record wasn’t being saved, and i couldn’t work out why.  It wasn’t failing validations.  If i tried to do it manually in the console, with the same data, i got this odd situation:

@user.valid?
=> true

@user.save
=> false

@user.errors.full_messages
=> []

WTF? i thought…  after a lot of logging, i managed to work out that it was because i’d added a before_create filter, which set one of the fields to the same as whatever was in a particular field in an associated object, like this:

before_create :set_student_access_to_same_as_music_service</pre>
def set_student_access_to_same_as_music_service
  self.student_access = self.music_service.show_student_access if self.music_service
end

So, if the user’s music service’s show_student_access field is false, we set the new user’s student_access to false, as a sensible default value.  As a side effect, the method returns false, as that’s the last thing to be evaluated.  And because the callback method returned false, the callback chain is halted and the record isn’t saved.  But, it’s not failing validations so i don’t get to see why.

Now, i’m not saying this is wrong – it’s in the api after all.  I’m just curious as to how many people were also unaware of this – out of the people i’ve spoken to it seems about only half of them knew, which is pretty bad considering it’s a major feature of callbacks, that would be causing unexpected effects all over the place if you weren’t careful about not returning false.  So, would you mind answering my poll, below?

thanks! max

Making acts_as_ferret deal nicely with pluralisation

I recently discovered how to nicely deal with pluralisation in the Rails plugin ‘acts_as_ferret‘ (which we use in our site). When i search for ‘trumpet’, i want to find resources with ‘trumpets’, and vice versa. By default acts_as_ferret doesn’t do this, but it’s easy to add. I found the answer in this post on the acts_as_ferret mailing list/forum:

http://www.ruby-forum.com/topic/80178

Anyway, here’s how i used it.  I’m going to assume that anyone reading this has already installed the ferret gem and acts_as_ferret plugin.

Make a new file in lib, called ‘acts_as_ferret_extensions.rb’

#lib/acts_as_ferret_extensions.rb

module Ferret::Analysis
  class StemmingAnalyzer
    def token_stream(field, text)
      StemFilter.new(StandardTokenizer.new(text.downcase))
    end
  end
end

Now require this file in the place in your config where you require stuff.  In the default case this would be in config/environment.rb:

#in config/environment.rb
require 'acts_as_ferret_extensions'

Next, change your calls to acts_as_ferret to specify an analyzer: here’s one of mine as an example, from the ‘Asset’ class in this case.

  #in asset.rb (an example from my app)
  acts_as_ferret( :ferret => {:analyzer => Ferret::Analysis::StemmingAnalyzer.new},
                  :fields => { :name => {},
                               :description => {}
                               #...etc
                              } )

Now, delete the index and rebuild it – deleting it might not be necessary but i’ve found weird results rebuilding after changing the options to acts_as_ferret: deleting the index is definitely safer.

#at the command line
rm -r index/development/asset/*
#in the console
Asset.rebuild_index

Now, when you do a search, specify this analyzer again:

#in the console
assets = Asset.find_with_ferret("trumpet", :page => 1,
                                           :per_page => 20,
                                           :analyzer => Ferret::Analysis::StemmingAnalyzer.new)

And that should be it!

Creating a custom “be_valid” matcher for rspec_on_rails

I’ve been discovering the excellent ‘behaviour-driven development’ plugin ‘rspec’ , and its rails-specific cousin (nephew?) ‘rspec_on_rails’ this week.

At the moment i’m testing that validation works properly on some models. Now, rspec already generates a matcher for any predicate method that can be called on an object (eg “object.should be_foo” for “object.foo?.should eql(true)”). I was using this with valid?, which we use on an typical (ie ActiveRecord) model to see if it will pass validation (and therefore be able to be saved to the database). This is generally nicer than trying to save it and seeing if it saved ok.

So, i was writing specs like this

it "should validate with only a name" do
  @foo = Foo.new(:name => "bar")
  @foo.should be_valid
end

This works, in terms of correctly passing or failing, but it doesn’t give me much information – it just tells me that something fails validation. I could get at the error messages in the test by doing the test as follows:

it "should validate with only a name" do
  @foo = Foo.new(:name => "bar")
  @foo.should be_valid
  @foo.errors.full_messages.should eql([])
end

but this is a bit tiresome. I’d rather just write “should be_valid”. So, i decided to write a custom matcher that overrides the auto-generated ‘be_valid’ method and tells me why it failed validation (if it does fail). Here’s how i did it. Thanks to the excellent peepcode rspec movies for the inspiration and knowhow!In my spec folder, i made a file called ‘custom_matcher.rb’. In it is a module which will hold any custom rspec matchers i write, though there’s only one at the moment.

module CustomMatchers
  #checks that an AR model validates, by testing error messages from .valid?
  #displays any error messages recieved in test failure output
  class BeValid
    #do any setup required - at the very least, set some instance variables.
    #In this case, i don't take any arguments - it simply either passes or fails.
    def initialize
      @expected = []
    end

    #perform the actual match - 'target' is the thing being tested
    def matches?(target)
      #target.errors.full_messages is an array of error messages produced by the valid? method
      #if valid? is true, it will be empty
      target.valid?
      @errors = target.errors.full_messages
      @errors.eql?(@expected)
    end

    #displayed when 'should' fails
    def failure_message
      "validation failed with #{@errors.inspect}, expected no validation errors"
    end

    #displayed when 'should_not' fails
    def negative_failure_message
      "validation succeeded, expected one or more validation errors"
    end

    #displayed in the spec description if the user doesn't provide one (ie if they just write 'it do' for the spec header)
    def description
      "validate successfully"
    end

    # Returns string representation of the object being tested
    def to_s(value)
      "#{@errors.inspect}"
    end
  end

  # the matcher method that the user calls in their specs
  def be_valid
    BeValid.new
  end
end

#To hook it up, add the following require to spec_helper.rb:
require 'spec/custom_matchers'

#And add the following line to the "Spec::Runner.configure do |config|" section:
config.include(CustomMatchers)

This is about as simple as a matcher could be – i don’t take any arguments at all, so the expected result is always an empty array (of errors). The ‘value’ added by this is that i’m now outputting the error messages (if any) in the failure report. Because i’ve provided a description method, i don’t even need to provide a description to my specs if i don’t want to: for example, here’s a test which will fail (and its surrounding description):

describe SchoolSubscriber, ", when we make an empty school_subscriber, " do
  before do
    @ss = SchoolSubscriber.new
  end

  it do
    @ss.should be_valid
  end
end

And here’s the failure report from rspec:

1)
'SchoolSubscriber , when we make an empty school_subscriber,  should validate successfully' FAILED
validation failed with ["Phone number can't be blank", "First name can't be blank", "Position can't be blank", "School can't be blank", "Last name can't be blank", "Email can't be blank"], expected no validation errors

How to quickly wipe and recreate a db table that doesn’t have a migration

Sometimes a table gets messed up and the best thing to do is to wipe it and recreate it from scratch.  Also sometimes, we don’t have a rake migration for it, or it was built up through a series of messy migrations that add bits, remove them, add other stuff, etc.

If we delete all the rows, we DON’T always get the same as an empty table – if, like in most rails tables, we have a primary index (usually id) which is set to autoincrement, if we delete all the records and then add one, the first row won’t be numbered 1, it will be numbered after the last row to be created.  So, we’re not really starting from scratch.

A nice quick way to just recreate a blank version of the table from scratch is as follows.

1) Make an sql dump of the entire database:  at the command line, type

rake db:structure:dump

2) This will create an sql script file called development_structure.sql in the db folder, which contains instructions for adding all of the tables to the db.  Open it in a text viewer and look for the section concerning the table you want.  Copy it out.

3) Lose the existing table – at the sql command line type “drop table your_table_name”

4) In a sql browser, or at the command line, paste in the section of sql script from the dump and run it.

Voila – nice empty table, with indexes all starting from scratch.

5) If you have any migrations to populate the table with data, you can copy the code out of the ‘up’ method, run a rails console, paste in the code and run it.

Split a class (with existing data) with single table inheritance

Let’s say that i have a table called Person, with a corresponding database table ‘people’, that already has a lot of records in it. It so happens that in my app men and women have slightly different rules associated with them, and are often searched seperately (eg search boxes have a ‘only search women’ checkbox).

The database represents gender by a field called ‘male’, which holds a boolean: men are ‘true’ and women are ‘false’. (this seems a bit sexist, but it ensures that there are only two possible values).

I’m sick of typing

:condition => ["male = ?", person.male]

all the time, and i’m generally sick of dealing with Person when there really should be two seperate classes which are dealt with differently. I want instead to be able to deal with Man and Woman, eg to sayMan.find(:all) etc, but i don’t want to start making serious changes to the database schema.

We can fix this with single table inheritance. This has two stages: 1) alter the database, and 2) extend Person into two new classes.

Stage 1 – we create a new string field in the ‘people’ table called ‘type’. This is simple with a database migration: at the command line (making sure we’re inside our project directory), type

ruby script/generate migration add_type_to_people

This should create a migration with that name.

Now, this migration has to do a bit more work than usual. We’re adding a new column, but we need to populate it as well, with either “Man” or “Woman”, which we don’t normally do in migrations. There’s nothing magical happening in migrations – we can put any rails command in there and it will get executed when the migration is run. In this case, we want to do an SQL update on all the records, and change the value in ‘type’ depending on the value of ‘male’. As it happens there’s a nice rails helper for this, called ‘update_all’. This allows us to enter the new value and an optional condition, without having to use SQL.

Looking at the ‘male’ field, we see that it happens to default to true (Come and see the sexism inherent in the system! Joking aside, if you’re going to default a boolean field, always default it to 0 (false) so that NULL and false are sort of equivalent.  Anyway lets assume that whoever set this table up hadn’t had their morning coffee yet). That means that while a woman has ‘male = false’, a man might have ‘male = NULL’. With our calls to update_all, we should follow this logic, and first set all of the values of ‘type’ to ‘Man’, then set it to ‘Woman’ where ‘male = false’. So the calls to update_all will be -

Person.update_all(‘type = “Man”‘)

(note the quotes here – single around the whole argument, double around the string “Man”)

Person.update_all(‘type = “Woman”‘, ‘male = 0′)

Note that we compare male against ‘0′, not ‘false’. This is because the database (well mysql at least) stores bools as 0 and 1 values in a ‘tinyint’ field.

So, here’s the whole migration:

class AddTypeToPeople < ActiveRecord::Migration
def self.up
add_column :people, :type, :string
Person.update_all(‘type = “Man”‘)
Person.update_all(‘type = “Woman”‘, ‘male = 0′)
end

def self.down
remove_column :people, :type
end
end

Then run it with (at the command line)
rake db:migrate

That should be it for stage 1. We can check by looking in the database, either using a gui (such as heidisql) or an sql request like

select male, type from dbname.people

which will show us only the male and type columns, which we can look down to see that the values match up properly.

Stage 2 – This is easier – first we need to add two new models, Man and Woman, which extend Person. Super simple.

in app/models/man.rb

class Man < Person
end

in app/models/woman.rb

class Woman < Person
end

That’s it! We can now call methods on Man and Woman as we would on Person.

Actually, we’re not quite out of the woods yet – what happens when a new person is added to the database, or when someone whose gender was left blank is updated to be female? (ie male = false) We need to do a callback, that calls a method to correctly set the value of ‘type’ whenever a record is updated or created. For our purposes, before_save is fine.

So, in person.rb -

class Person < ActiveRecord::Base

#will be triggered when saving a new or updated object
before_save :set_type

#and now the method that the callback calls. Remember that male
#defaults (bizarrely) to ‘true’
def set_type
if self.male == false
self.type = “Woman”
else
self.type = “Man”
end
end

#or, for those who like conditional expressions -
def set_type
self.type = (self.male == false ? “Woman” : “Man”)
end

…
…
end

This should keep our database in line.