Fo’ Shizzle

Short story: When you’ve got a module doing this:

module Taglist
  def self.included(base)
    base.class_eval do
      [:save, :save!].each do |method|
        alias_method_chain method, :taglist
      end
    end
  end
...

Ensure that you protect against including it twice, like so:

class BuildJob < ActiveRecord::Base
  include Taglist unless self.include?(Taglist)
...

Why? Because otherwise you'll waste hours trying to understand why your code works fine in development but starts failing with StackLevelTooDeep errors when you run rake tests.

Bleh.

Well, maybe fun, anyhow. Possibly profit.

One of the reasons I had put off doing any real testing of this Rails project was that I didn’t feel like dealing with figuring out how to avoid messing with the SLURM stuff when doing testing. I really don’t want to kick off fourty real build jobs when I run my tests, but I really should be testing all the goop that leads up to and follows after the interaction with SLURM.

The answer is so simple that I didn’t believe it at first. All I needed was to drop a file in test/mocks/test with the same name as my SLURM library (happens to be slurm.rb). In this file, I simply require the original library, and then stub out the run_slurm_command subroutine. Et voila! I can now safely pretend to interact with the SLURM system without actually doing anything to it.

Incidentally, this works just fine for modules. In other words, it will work regardless of what you are mocking. I make this note because after reading the documentation I had the impression that it would only work for mock objects. My SLURM library is just a collection of subroutines, so I was thinking that I needed to make it a class. In fact, the Mock infrastructure is just Ruby’s built-in metaprogramming — I can open up an existing module or class, monkey around with its innards, and then send it on its way. Sweet!

OK, OK… I’ll admit it. My first real Rails app, and I ignored testing.

sloccount tells me that I have 1,536 LOC in my app/ subdir, and until today, I had no working unit tests. *hangs head in shame*

Yesterday I was working on allowing my app to receive cvs-commits emails, so that I can kick off builds and tests for monitored CVS branches. I realized after a while that what I was doing in the Rails console was essentially re-typing unit tests, over and over. Even worse, I was then throwing them away when I was finished! Having never really worked with the Rails testing framework, I had basically ignored it because I felt I didn’t have time to do testing.

Fortunately, I haven’t been bitten too badly by my earlier decision, but after yesterday and today I’ve realized that once you get past a few hundred LOC, you just can’t make changes and trust that you’ve remembered what those changes will affect.

Now that my project has come out of skunkworks mode, I need to make sure that it’s always production-worthy. I’ve got executive buy-in now, so I’ll take the time to ensure that obvious bugs don’t creep in while I’m hacking. In the future, I’ll try to be a bit more test-driven, since it’s basically how I work anyhow. I’ve just been doing it all by hand.


Started
..........................................................
Finished in 0.907775 seconds.

58 tests, 91 assertions, 0 failures, 0 errors


It’s a good start. Now I’ve got to work on functional and integration testing.

I’d like to say a bit about how cool ActiveRecord Observers are.

I am writing a highly event-driven app in Rails (e.g. create a BuildJob, and kick off a SLURM job for said job). Now, the simplistic way to handle this might be to write a daemon that sits in a loop and watches for new BuildJob records. In fact, that’s essentially how the old implementation worked (I didn’t write it!).

As I was writing the application (and learning Rails at the same time), I discovered ActiveRecord::Observer. I was thunderstruck. Folks who have been programming “for real” for a while may laugh here, but this was the first time I’d seen a good use of the Observer pattern that I’d learned about way back when. I’ve been a sysadmin for most of my career, though, so cut me some slack.

Anyhow, instead of a clunky poll-based design, I now have a nice, clean, and logical design that tries to create a SLURM job whenever a new BuildJob is created. If the SLURM job creation fails, so does the BuildJob creation, things are rolled back in the DB, and all is safe.

That’s the kind of thing that makes programming really fun and gratifying. I love finding elegant solutions that solve multiple problems at the same time.

Oh, you want to see some code? Ok.

class BuildJobObserver < ActiveRecord::Observer
  include Slurm
  include LTS

  # We want to try to create a SLURM job before saving the BuildJob
  # record.  This way, we know that a BuildJob always has an associated
  # SLURM job.  The problem is that we don't yet have an id for the BuildJob
  # when we try to create the SLURM job, and so we can't tell the SLURM
  # job which BuildJob it's associated with.  The solution is to create
  # the SLURM job, but don't let it run until it's updated with the
  # BuildJob's id.
  #
  # before_create
  #   * create slurm job with status "held" (use SchedulerType=sched/wiki)
  #   * save slurm job id in lbats job
  # after_create
  #   * update slurm job name with lbats id for future linkage
  #   * update slurm job with priority > 0 to enable it

  def before_create(job)
    batch_options = {
      :command   => lbats_job2lts_build_cmd(job),
      :name      => "lbats_need_id",
      :features  => job.architecture.name,
      :partition => "#{job.architecture.name}_build",
    }

    job.logger.debug "command: #{batch_options[:command]}"

    if slurm_id = slurm_submit_batch_job(batch_options)
      job.resource_manager_id = slurm_id
      job.logger.info "Created new SLURM job: #{slurm_id}"
    else
      # So here's a problem, maybe...  If we submit 5/6 build jobs, and
      # the 6th fails, what about the 5 SLURM jobs that were submitted OK?
      # If we bomb out here, the BuildJobs won't be created in the DB, which
      # is good, but we've still got 5 SLURM jobs that are unassociated.
      # Not sure there's much we can do here but cross our fingers.  :/
      raise "Unable to create SLURM job"
    end
  end

  def after_create(job)
    slurm_update_job_attributes(job.resource_manager_id,
                                {:priority => job.build_request.priority.value,
                                 :name => "lbats_build_#{job.id}"})
  end

  def before_update(new_job)
    old_job = BuildJob.find(new_job)

    # check for a state change
    if old_job.job_state != new_job.job_state
      # tell the request about the state change
      new_job.build_request.register_job_state_change(new_job)
      # TODO: sanity-checking of all state transitions?
      case new_job.job_state
        when JobState.Canceled
          # NB: If the job is canceled outside of LBATS (i.e. someone
          # runs scancel), we'll be doing a double-cancel because
          # the jobcomp script will post an update to the LBATS job
          # and bring us here.  I think this is OK, because trying
          # to scancel an already-canceled job is not fatal.
          slurm_cancel_job(new_job.resource_manager_id)
      end
    end
  end

  def after_save(job)
    # tickle the build request's updated_at
    job.build_request.save
  end

  def before_destroy(job)
    # make sure we nuke the job from SLURM, too!
    slurm_cancel_job(job.resource_manager_id)
  end
end