Jekyll2015-09-14T18:51:33-04:00http://wegowise.github.io/The WegoWise Development BlogThe WegoWise teamA Memcached Trick and an Old Watchdog2014-10-09T16:56:00-04:002014-10-09T16:56:00-04:00http://wegowise.github.io/blog/2014/10/09/a-memcached-trick-and-an-old-watchdog<p>This article describes some code we developed to fix an issue with our Memcached
environment. This took the form of adding a method to a class that we didn't
own. The article then describes the way that we used the <a href="https://github.com/wegowise/watchdog">Watchdog gem</a>
to ensure that our class extension won't cause problems further down the road.</p>
<h3>The problem</h3>
<p>We use Memcached both for page fragment caching and for caching large
collections of objects that require significant work to generate and fetch. For
example, we have a page where we need to display a dot on a scatter plot for
every building owned by the user. Associated with each dot is a small amount of
work to 1) get associated data out of the database and 2) perform calculations
on the data to get derived values that are used in the display of the scatter
plot. This work is negligible for small numbers of buildings, but for users with
thousands of buildings the work adds up to unacceptably long request times
(actually they hit our self-imposed server timeouts first). For these larger
accounts we simply can't afford to perform all those database queries and
calculations during the server request cycle.</p>
<p>{% img right /images/scatterplot.png /images/scatterplot.png 400 358 "A scatter plot showing building values" %}</p>
<p>Instead we cache all that work server-side so that a request for the same data
can be served up directly without the need to fetch most data from the database
or perform any calculations. We also employ an aggressive <strong>cache-warming</strong>
strategy so that data is usually pre-cached when a user requests it.</p>
<p>There's much more to say about how to implement effective caching in an
enterprise Rails application. Here we just want to talk about a specific pitfall
we ran into with our caching setup.</p>
<p>For our caching we use the built-in caching mechanism in Rails along with the
<a href="https://github.com/mperham/dalli">dalli gem</a>, which provides a Memcached caching store called
<code>ActiveSupport::CacheDalliStore</code>. This allows us to cache objects using the
following simple API:</p>
<pre><code class="ruby">def meaning_of_the_universe
a_bunch_of_work_to_calculate # => 42
end
Rails.cache.fetch('meaning_of_the_universe') { meaning_of_the_universe } # => 42
</code></pre>
<p>The way this works is that if the key "the_meaning_of_the_universe" is found in
Memcached (called a <strong>cache hit</strong>) then we can skip the expensive calculation
contained in the block. If the key isn't set (called a <strong>cache miss</strong>) then we
do the work but then send the value to Memcached so that we can avoid the work
the next time around. We can also do <code>Rails.cache.read</code>, which will read the
value but do nothing if there is a cache miss, and <code>Rails.cache.write</code>, which
will write the value without checking first to see if it already exists.</p>
<p>Every time you check the value of a cache, or write to it, you are making a
request to a Memcached server (and a cache miss followed by a write will
actually be two requests). Unless your Memcached server is running on the same
machine as your application server there is going to be some network overhead
associated with performing a lookup. The more keys you check for the more that
network IO will add to the time of your request, which will eventually defeat
the purpose of having a cache in the first place.</p>
<p>To avoid this it can make sense to perform a <code>Rails.cache.read_multi</code> operation,
where you pass a list of keys and get back a hash of keys and values, with the
value set to <code>nil</code> where there was a cache miss. This was the strategy we took
on our page with the thousands of buildings displayed on a scatter plot. The
code looked (very roughly) something like this:</p>
<pre><code class="ruby">buildings_hash = Rails.cache.read_multi(*building_keys)
buildings_hash.each do |key, value|
buildings_hash[key] = Rails.cache.fetch(key) unless value
end
</code></pre>
<p>That is, you perform a bulk read on the keys and then go back and fill in the
ones that were missing, instead of checking each one.</p>
<p>In general this strategy has worked out well for us. The problem (finally) that
we ran into is that with our particular hosted Memcached provider <strong>and</strong> very
large accounts we would get strange behavior. Sometimes the page would load
right away and other times it would take minutes to generate, with no rhyme or
reason. When we looked into things it turned out that <em>sometimes</em> the
<code>#read_multi</code> operation was failing and we were treating the result as an empty
hash. This meant that the page had to go recalculate and write the values for
all of those buildings, even though the values actually existed in Memcached.</p>
<h3>The solution</h3>
<p>For whatever reason our hosted Memcached provider couldn't handle the large bulk
read request we were sending them. The obvious solution was to break up the
request into smaller chunks. To this end we created the following logic:</p>
<pre><code class="ruby">def read_multi_chunked(*cache_keys, chunk_size: nil)
return {} if cache_keys.empty?
chunk_size ||= ENV.fetch('MEMCACHED_READ_MULTI_CHUNK_SIZE') { 1_000 }
chunked_keys = cache_keys.in_groups_of(chunk_size, false)
chunked_keys.reduce({}) do |hash, chunk|
result = begin
Rails.cache.read_multi(*chunk)
rescue Dalli::RingError
{}
end
hash.merge(result)
end
end
</code></pre>
<p>A quick explanation of the code:</p>
<ul>
<li>take in a list of cache keys</li>
<li>get the chunk size from a parameter or from an environment variable</li>
<li>chunk the keys</li>
<li>perform <code>read_multi</code> operations for each chunk</li>
<li>combine the results into a single hash</li>
</ul>
<p>In practice this new method has the same method signature (when the chunk size
parameter is omitted) and output as the original <code>read_multi</code> method.</p>
<h3>Next problem</h3>
<p>The only problem remaining was deciding where to put this method. We try to keep
methods out of the global scope, so we wanted to put it on some logical module
or class. We discussed briefly the possibility of overriding
<code>ActiveSupport::DalliStore#read_multi</code> and using <code>alias</code> trickery to call the
original <code>read_multi</code> inside the new implementation, but decided against it on
moral grounds. We decided, however, that it would make sense to put the method
on <code>ActiveSupport::DalliStore</code> as an additional method, so that any code that
was previously using <code>Rails.cache.read_multi</code> could just switch over to using
<code>Rails.cache.read_multi_chunked</code>. In other words we could do this:</p>
<pre><code class="ruby">module ActiveSupport
module Cache
class DalliStore
def read_multi_chunked(*cache_keys, chunk_size: nil)
return {} if cache_keys.empty?
chunk_size ||= ENV.fetch('MEMCACHED_READ_MULTI_CHUNK_SIZE') { 1_000 }
chunked_keys = cache_keys.in_groups_of(chunk_size, false)
chunked_keys.reduce({}) do |hash, chunk|
result = begin
read_multi(*chunk)
rescue Dalli::RingError
{}
end
hash.merge(result)
end
end
end
end
end
</code></pre>
<p>(Note that we can drop the reference to <code>Rails.cache</code> and just use <code>read_multi</code>
now, since an instance of <code>DalliStore</code> is what <code>Rails.cache</code> referred to
previously).</p>
<p>We liked this because it allowed us to look for our <code>read_multi_chunked</code>
operation in a familiar place, but we were also aware that we had opened us up
to a somewhat unlikely but nevertheless troubling risk that is inherent in all
such extensions: what if the maintainers of Dalli decided that they needed a
method that could handle chunked <code>read_multi</code> operations and what if they
implemented a method with the same name? We wouldn't necessarily notice the
change when we updated the gem. There might be subtle differences or
implications of their implementation that could affect our application. What if
the maintainers updated internal code to use their new method, which would now be
using our own implementation instead?</p>
<p>That might sound like a lot of ifs and in this case it's hard to think of a way
that there would be a catastrophic effect (since it's unlikely that
"read_multi_chunked" could mean something radically different), but it
highlights the general issue with creating extensions on classes that you don't
own. If nothing else, you might want to know about and use the maintainer's
version of the method because it could be more efficient and you can stop
maintaining your own version.</p>
<h3>The extended solution</h3>
<p>WegoWise maintains and uses a gem called
<a href="https://github.com/wegowise/watchdog">Watchdog</a>
(written by a former Wegonaut, <a href="https://github.com/cldwalker">Gabriel Horner</a>)
that you can use to put an end to all this fretting about hypotheticals
involving collisions with upstream code. By now it's a pretty old dog (last
updated in 2011) but that's because what it does is fairly simple. Watchdog will
check to make sure that your modification to a class doesn't collide with any
existing methods. It performs this check at runtime so the moment upstream code
creates a method that you are extending you will get an exception.</p>
<p>To use Watchdog you need to convert your extension into a mixin module. This is
necessary because Watchdog needs to hook into the <code>extend_object</code> callback
method that is run when <code>extend</code> is called on a class. This is what our code
looks like with Watchdog:</p>
<pre><code class="ruby">module Ext
module ActiveSupport
module CacheDalliStore
extend Watchdog
def read_multi_chunked(*cache_keys, chunk_size: nil)
return {} if cache_keys.empty?
chunk_size ||= ENV.fetch('MEMCACHED_READ_MULTI_CHUNK_SIZE') { 1_000 }
chunked_keys = cache_keys.in_groups_of(chunk_size, false)
chunked_keys.reduce({}) do |hash, chunk|
result = begin
read_multi(*chunk)
rescue Dalli::RingError
{}
end
hash.merge(result)
end
end
end
end
end
module ActiveSupport
module Cache
class DalliStore
include Ext::ActiveSupport::CacheDalliStore
end
end
end
</code></pre>
<p>The new version is certainly more verbose, but we think that's acceptable.</p>
<p>If it wasn't clear by now, we are okay with occasionally modifying classes that
we don't own. Sometimes monkeypatching (or <a href="https://vimeo.com/17420638">freedom patching</a>!)
can be the most elegant solution to a problem, especially when you want to make
a change to an interface that is called in many places throughout your
codebase.</p>
<p>That said, we don't think that class modification should be your first, or even
second, option when thinking about a software problem. There are usually better
approaches, such as creating a wrapper class that you can define your own
methods on, or creating a subclass of the class that you want to modify. If,
after careful consideration, you still decide that you want to modify that
class, we think it's reasonable for there to be some ceremony and cruft
involved.</p>
<p>Watchdog makes extending an upstream class considerably less dangerous, for
those occasions when a patch seems like an appropriate option.</p>
joeThis article describes some code we developed to fix an issue with our Memcachedenvironment. This took the form of adding a method to a class that we didn'town. The article then describes the way that we used the Watchdog gemto ensure that our class extension won't cause problems further down the road.Great new feature in RSpec 3: verifying doubles2014-09-03T13:23:00-04:002014-09-03T13:23:00-04:00http://wegowise.github.io/blog/2014/09/03/rspec-verifying-doubles<p>We recently upgraded our test suite to RSpec 3. We did this to stay up to date
with the most recent code rather than because we wanted any particular new
feature. And for the most part RSpec 3 doesn't seem to add many features so much
as it cleans up the API (the biggest change is that you can now use
<code>expect(something)</code> syntax everywhere, instead of <code>something.should</code>). In fact,
a lot of the changes that we had to make when upgrading involved adding gems
that had been <em>extracted</em> from RSpec.
<a href="http://myronmars.to/n/dev-blog/2014/05/notable-changes-in-rspec-3">This post</a>
by RSpec core team developer Myron Marston is a comprehensive summary of the
changes.</p>
<p>There is one new feature in RSpec 3 that introduces a new and powerful behavior
that has important implications for testing practice. I would go so far as to
say that the project seriously buried the lede on this. The new feature is
called <strong>verifying doubles</strong>, and in a nutshell it allows you to create test
doubles that verify themselves as conforming to the same interface as a "real"
object. Thanks to <a href="http://xaviershay.com/">Xavier Shay</a> for adding this great
feature.</p>
<p>There are two steps for gaining access to this magic. The first is by using
<code>instance_double</code> (and the related <code>class_double</code>) instead of <code>double</code>
throughout your code. The second is to enable <strong>verifying partial doubles</strong> in
your test suite.</p>
<h3>Using instance_double in your code</h3>
<p>Here is an example of using <code>instance_double</code> in a spec. Note that you need to
pass the class that the double is meant to represent as the first argument to
<code>instance_double</code>:</p>
<pre><code class="ruby">class Blog
attr_accessor :posts
def descriptions
posts.map(&:description)
end
end
class Post
attr_accessor :title, :author, :body
end
# the test
require 'spec_helper'
describe Blog do
specify '#descriptions returns descriptions from posts' do
post = [instance_double(Post, description: 'My great post')]
blog = Blog.new(posts: [post])
blog.descriptions.should eq(['My great post'])
end
end
</code></pre>
<p>When you run this test you will get the following error:</p>
<pre><code class="text">Failure/Error: post = [instance_double(Post, description: 'My great post')]
Post does not implement: description
</code></pre>
<p>The verifying double alerted us to the fact that we hadn't implemented
<code>#description</code> on <code>Post</code>. Hopefully this bug would have been caught further down
the line in an acceptance test but there are often gaps in coverage.</p>
<p>Even better, verifying doubles will verify the <strong>number of arguments</strong> on a
method:</p>
<pre><code class="ruby">class Invoice
attr_accessor :total
def add_line_item_to_total(line_item)
self.total ||= 0
self.total += line_item.amount
end
end
class LineItem
attr_accessor :amount
def amount(include_cents)
include_cents ? amount.to_f : amount.to_i
end
end
# the test
require 'spec_helper'
describe Invoice do
specify '#add_line_item_to_total adds line item amount to total' do
line_item = instance_double(LineItem)
line_item.stub(:amount).and_return(10)
invoice = Invoice.new
invoice.new.add_line_item_to_total(line_item)
invoice.total.should eq(10)
end
end
</code></pre>
<p>This results in a different error:
<code>text
Failure/Error: self.total += line_item.amount
ArgumentError:
Wrong number of arguments. Expected 1, got 0.
</code></p>
<p>Just in case you missed it, RSpec is telling us that we are using the wrong
number of arguments to call a method <strong>on the double</strong>. This is really powerful
when you think about how we tend to extend method signatures. First we start off
with a method call without arguments. Then a little later we decide we need more
flexibility and add a switch flag like <code>include_cents</code>. If our test suite uses a
lot of doubles we might not realize that we've just broken a lot of code (in the
real world we would probably be more careful and make <code>include_cents</code> an
optional argument).</p>
<p>You can also use <code>class_double</code>, which works the same as <code>instance_double</code> but
for classes.</p>
<p>(<strong>Update</strong>: see Myron Marston's comment below about other ways that the method
signature is verified)</p>
<h3>Enabling verifying partial doubles</h3>
<p>A <strong>partial double</strong> is what is more commonly known as a stubbed method. It is
arguably even more important that stubbed methods on real objects correspond to
reality, especially when you are stubbing code for external libraries that you
do not test directly:</p>
<pre><code class="ruby">class FileUploader
def upload(file)
ExternalLibrary.upload(file)
end
end
describe FileUploader do
specify '#upload sends the file to External Service' do
ExternalLibrary.should_receive(:upload).with('somefile.csv')
FileUploader.new.upload('somefile.csv')
end
end
</code></pre>
<p>I haven't included the code for <code>ExternalLibrary</code> because the point is that we
don't own this code and so we tend to be less sure about the interface. What if
we update the gem version and the method signature changes? As written this test
will keep on happily chugging along, even though the code will throw errors in
production.</p>
<p>I call this situation when your code is broken but your test suite is all green
due to use of stubs a <strong>La La Land</strong>. To avoid a La La Land you can enable
RSpec's verify partial doubles feature:</p>
<pre><code class="ruby">RSpec.configure do |config|
config.mock_with :rspec do |mocks|
mocks.verify_partial_doubles = true
end
end
</code></pre>
<p>Now your stubs will behave like verifying doubles, throwing an error if you try
to stub a method that doesn't exist on the object, or if you stub a method with
the wrong number of arguments.</p>
<p>I presented this step second because I wanted to explain verifying doubles
through examples using <code>instance_double</code> first, but you probably want to enable
this option first. After that you can go through your test suite replacing
<code>double</code> with <code>instance_double</code> at your leisure. You might find it eye-opening
when you turn on verifying partial doubles. Suddenly a lot of tests that you
thought were sound might turn out to contain stubs on methods that don't exist,
or that take different numbers of arguments. It can be an opportunity to find
out what your test suite is really testing.</p>
<h3>Gotchas</h3>
<p>There are two gotchas involving verifying doubles that we've run into so far.</p>
<p>The first involves redefinition of <code>#respond_to?</code>. Under the hood RSpec uses
<code>#respond_to?</code> to determine if it should throw an error when a verifying double
tries to implement a method. Sometimes you'll find that certain classes redefine
<code>#respond_to?</code> (for example if they lean on <code>#method_missing?</code> to respond to
undefined method calls), but they don't always implement it correctly, leaving
off the optional second argument. This can cause an <code>ArgumentError</code> when
stubbing on these classes. For example, this used to be an issue with
<code>Rails::Railtie::Configuration</code> but it appears to be fixed in recent releases.
To be clear, this was never a bug in RSpec but an issue of classes incorrectly
implementing <code>#respond_to?</code>.</p>
<p>The second issue is more endemic. Classes that uses tricks to defer defining
methods will cause your verifying doubles to fail if an actual instance of the
class hasn't been created and put through its paces first. This can be
frustrating when working with <code>ActiveRecord</code> models, for instance, since they
use a ton of fancy introspection on database columns to define methods on the
fly. Depending on the order of your tests you can end up with false positive
errors where RSpec will complain that a method doesn't exist on the class.
Merely using the method on a real instance in your test anywhere prior to the
stub request will silence the error, however. We don't have a good solution for
this problem, so we've been avoiding using <code>instance_double</code> on <code>ActiveRecord</code>
models for the time being, instead using <code>mock_model</code> from the
<a href="https://github.com/RSpec/RSpec-activemodel-mocks">RSpec-activemodel-mocks</a> gem.
Something you can try is to create in-memory instances of your models before
your test suite runs. (<strong>Update</strong>: in the comments Myron Marston points out
that you can use <code>object_double</code> with an instance of the object)</p>
<h3>Some testing theory</h3>
<p>You can skip this section if you just wanted to know how to use verifying
doubles. That's all done.</p>
<p>There have recently been some (in my opinion) rather productive debates in the
Ruby world about the role of mocks and stubs in testing. On the one side were
the proponents of mockist testing as a component of Test Driven Development
(TDD). On the other side was the argument that mockist tests are poor tests that
produce (in my terms) dangerous La La Lands where all the code is small and
"testable" but doesn't necessarily work together. The anti-mockist argument
further went that if you can get your tests to run "fast enough" (since another
argument for using doubles is that they have less overhead) then you're better
off using doubles much less frequently and simply exercising your real code.</p>
<p>Something that seemed to get lost in that debate is the role that doubles play
in the most important aspect (he asserted) of Ruby programming, which is duck
typing. When you write a method definition that takes in an object, like</p>
<pre><code class="ruby">def add_line_item_to_total(line_item)
self.total ||= 0
self.total += line_item.amount
end
</code></pre>
<p>you may think that you wrote a method that takes in a <code>LineItem</code>, but you
actually wrote a method that takes in <strong>anything that quacks like</strong> a
<code>LineItem</code>, that is anything with an <code>#amount</code> method. Doubles in specs serve as
a form of documentation of the always implicit contract that methods require
incoming objects to uphold. They help keep the programmer honest by forcing them
to explicitly specify all the methods that an object is required to implement.
If you need to stub 10 values on a double to make a test pass, that should tell
you something about the degree of coupling in your code and should make you
wonder if your classes are following Single Responsibility Principle.</p>
<p>I'm excited about verifying doubles because they introduce something close to
the Interface concept that other languages have, but without the formality of
having to create an explicit definition. Interfaces as language constructs in
typed languages like Java are used to allow for more flexibility in passing
arguments and returning values. Instead of saying that a method can only receive
an argument of type <code>Widget</code> you can say that a method can receive any object
that implements the <code>Widget</code> <em>interface</em>, which has to be formally specified in
a separate class-like definition. Often the interface will be named <code>IWidget</code> to
distinguish it from the <code>Widget</code> class that <em>implements</em> <code>IWidget</code>. These
languages perform compile and runtime checks to make sure that classes correctly
implement the interfaces that they <em>claim</em> to implement. So if <code>Widget</code> claims
to implement the <code>IWidget</code> interface then it must implement all the methods
(with the correct number of arguments and argument types) that are specified by
<code>IWidget</code>, or else an error will be raised.</p>
<p>In essence, verifying doubles do an interface check in your test rather than at
runtime in your code. Instead of maintaining a separate formal interface
definition you can simply refer to the method signatures of a canonical class,
like <code>String</code> or <code>File</code>. You can also use verifying doubles to make sure that
calling code only uses the common interface for an inheritance structure of
classes, by passing in the abstract class to <code>instance_double</code>:</p>
<pre><code class="ruby">class Publication
def title
raise NotImplementedError, 'Subclasses must implement #title'
end
end
class Magazine < Publication
attr_accessor :title
end
class Book < Publication
attr_accessor :title, :summary
end
class Publisher
attr_accessor :publication_listings
def publish(publication)
self.publication_listings ||= []
self.publication_listings << "#{publication.title} -- #{publication.summary}"
end
end
# test
require 'spec_helper'
describe Publisher do
describe '#publish' do
it 'adds a publication to the publications list' do
magazine = instance_double(Publication, title: 'A book', summary: 'words')
publisher = Publisher.new
publisher.publish(magazine)
publisher.publication_listings.should eq(["A book -- words"])
end
end
end
</code></pre>
<p>This test fails because somebody extended the <code>#publish</code> method's contract
beyond what <code>Publication</code>'s interface promises. Note however that this test
would not help us detect that <code>Magazine</code> does not implement a particular
interface if we went back and implemented <code>#summary</code> on <code>Publication</code>. That is a
different problem which in other languages would be solved by, well, an
interface check. In her
<strong><a href="http://www.poodr.com/">Practical Object Oriented Design in Ruby</a></strong>
book Sandi Metz addresses this problem by using a set of shared tests for the
common interface that she mixes into the unit tests for each concrete class. It
would be nice to have an automated way of doing this that reuses the verifying
double mechanism.</p>
<h3>Conclusion</h3>
<p>Verifying doubles may sound intimidating, but they are easy to switch over to
and use today. Start by turning on verifying partial doubles and then gradually
switch over your existing doubles to use <code>instance_double</code>. Expect to have to
fix some tests, but take comfort in the fact that you're making your test suite
so much more rigorous and meaningful with a relatively small amount of work.
Verifying doubles make RSpec a much more powerful and relevant testing tool.</p>
JoeWe recently upgraded our test suite to RSpec 3. We did this to stay up to datewith the most recent code rather than because we wanted any particular newfeature. And for the most part RSpec 3 doesn't seem to add many features so muchas it cleans up the API (the biggest change is that you can now useexpect(something) syntax everywhere, instead of something.should). In fact,a lot of the changes that we had to make when upgrading involved adding gemsthat had been extracted from RSpec.This postby RSpec core team developer Myron Marston is a comprehensive summary of thechanges.Timezones Are Hard: a Presentation2014-08-25T23:05:00-04:002014-08-25T23:05:00-04:00http://wegowise.github.io/blog/2014/08/25/timezones-are-hard-a-presentation<p>In this post I will describe an interesting presentation that one of the
WegoWise team gave at work. But before I do that I need to give some context.</p>
<h3>Prologue</h3>
<p>Toward the end<sup id="fnref:1"><a href="#fn:1" rel="footnote">1</a></sup> of every week the WegoWise development team takes time out to
hold a small hour-long "conference". We call it, creatively enough,
Mini-conference, or just mini-conf. Mini-conf is a very informal, very optional
occasion for developers to present on various topics that are loosely related to
software development<sup id="fnref:2"><a href="#fn:2" rel="footnote">2</a></sup>. The purpose of Mini-conf is to share information or
techniques, and also simply to provide an occasion for us to hang out<sup id="fnref:3"><a href="#fn:3" rel="footnote">3</a></sup>. Our
team is completely remote and although we see each other on video chat during
standups and pairing sessions we don't otherwise get many opportunities to all
be "in the same room" for an extended period of time.</p>
<p>Here are some topics that have come up during mini-conf:</p>
<ul>
<li><strong>Jeff</strong> - Using JRuby, i.e. 'Software as a platypus'</li>
<li><strong>Janet</strong> - Notes from Burlington Ruby Conference</li>
<li><strong>Joe</strong> - Everything You Wanted to Know About Joe's Workflow but Were Afraid
to Ask</li>
<li><strong>Dan</strong> - Introduction to Compass: more than just a bunch of mixins</li>
<li><strong>Nathan</strong> - Learn this one weird trick to decrease your Vim
startup time 97.62%</li>
<li><strong>John</strong> - BEES! (a statistical analysis of the chance that a Git ref will
have the string 'bee' in it.)</li>
<li><strong>Barun</strong> - A primer on color theory</li>
<li><strong>Jed</strong> - Search & Replace Across Files in Vim</li>
</ul>
<p>As you can see, the topics range from information that is directly useful to our
daily work, to tips about useful tools and ways to improve our software
environments, to presentations about concepts from other technologies or domains
that can provide us with new perspectives. The topics aren't always completely
serious, but there is usually something that we can take away and apply. We will
often have two or three presentations per mini-conf, although sometimes one
presentation will take up the whole hour or go even longer.</p>
<p>What follows is the <strong><em>shortest</em></strong> presentation we have ever had at Mini-conf.</p>
<h3>The Presentation</h3>
<p>This presentation was given by <strong>Nathan</strong> and goes as follows:</p>
<blockquote><h3>Timezones are Hard</h3>
<p>... unless you use <code>Date.current</code> and <code>Time.current</code>.</p>
<p>The end.</p></blockquote>
<h3>Epilogue</h3>
<p>Despite its brevity, this was one of the more useful and memorable presentations
we've had at Mini-conf. Let me explain.</p>
<p>The issue this presentation addresses, if you are not familiar with it, is that
in Rails <code>datetime</code> columns are usually stored in the database as UTC and
converted to a local timezone that is set in the application configuration
files. If you save a new datetime to the database it will automatically be
converted from the application time back to <code>UTC</code>.</p>
<p>The problem comes in when the application server (or a developer's computer) is
in a different timezone than the one that is set for the application, for
example if the application is set to Eastern Time and the server is located
in Central Time. In this case if you use <code>Time.now</code> in your code you will
get an instance of <code>Time</code> with the time zone offset for Central Time. This
can lead to a variety of subtle problems, mostly to do with intermittent tests
but potentially leading to serious bugs in production as well. The worst of
these bugs involve <code>Date.today</code>, because <code>Date</code> does not know about timezones.
Consider this somewhat realistic accounting bug:</p>
<pre><code class="ruby">class Invoice < ActiveRecord::Base
def overdue?
(Date.today - issued_at.to_date) >= 10
end
end
# test file
describe Invoice do
let(:invoice) { create(:invoice) }
describe '#overdue?' do
it 'is overdue if 10 or more days have gone by' do
contract.issued_at = 10.days.ago
contract.overdue?.should eq(true)
end
end
end
</code></pre>
<p>So the issue here is that this test will fail between 11PM and midnight CDT, if
the computer running the test is in CDT and the application is configured for
EDT. Let's say it is midnight EDT on the 29th of August. This means that it is
11PM on the 28th for the computer in CDT. As a result the statement <code>Date.today</code>
will return the date as the 28th. <strong>But</strong> the statement 10.days.ago will use the
Rails application timezone, because these methods come from <code>active_support</code>. So
the <code>issued_at</code> date for the contract will be the 19th, but today's date will
be the 28th, and less than 10 days will have elapsed.</p>
<p>Or something.
<a href="http://monkeyandcrow.com/blog/reading_rails_time_with_zone/">Timezones</a>
<a href="http://www.elabs.se/blog/36-working-with-time-zones-in-ruby-on-rails">are</a>
<a href="http://danilenko.org/2012/7/6/rails_timezones/">hard</a>.
The point is that you can avoid this confusion in Rails code by just using the
active_support methods <code>Date.current</code> and <code>Time.current</code> instead of
<code>Date.today</code> and <code>Time.now</code> to always get the current time or date in the
application timezone.</p>
<p>Also, team-building is hard, especially when everybody is remote. One thing that
can help is to set aside some time during the week for people to hang out and
share what they've learned with their colleagues.</p>
<p>The end.</p>
<div class="footnotes">
<hr/>
<ol>
<li id="fn:1">
<p>Mini-confs were initially held at the end of the workday on Friday, but our team is highly distributed and this meant that some developers in places like, say, Spain, had to log into their computers late on a Friday night to participate.<a href="#fnref:1" rev="footnote">↩</a></p></li>
<li id="fn:2">
<p>Sometimes instead of a presentation we will just take an article or hot button issue and have a conversation for the hour.<a href="#fnref:2" rev="footnote">↩</a></p></li>
<li id="fn:3">
<p>Using, of course, Google Hangouts.<a href="#fnref:3" rev="footnote">↩</a></p></li>
</ol>
</div>
JoeIn this post I will describe an interesting presentation that one of theWegoWise team gave at work. But before I do that I need to give some context.The new face of WegoWise2014-04-09T19:53:00-04:002014-04-09T19:53:00-04:00http://wegowise.github.io/blog/2014/04/09/the-new-face-of-wegowise<p>We recently redesigned the WegoWise website and whilst doing so I approached
some common problems a bit differently than I have in the past. This article is
about how I tried out some new tools and techniques to create mockups, organise
my code and write breakpoints (for responsive web design).</p>
<p><img src="/images/wegostories.png"></p>
<h4>I ditched Photoshop in favour of <a href="http://www.bohemiancoding.com/sketch/" title="Sketch website">Sketch</a></h4>
<p>There are so many reasons why this made sense but here are a few
of the key ones (Disclaimer: Sketch is a Mac-only application):</p>
<h5>Price!</h5>
<p>Sketch costs $79 to download, Photoshop costs $239.88 for a year (based
on the price at the time of writing for one years subscription to the Creative
Cloud for one full application).</p>
<h5>Export to SVG option</h5>
<p>It’s a format that I think there’s no avoiding in modern
web design and for good reason.</p>
<h5>Small file sizes that can handle a lot of layers before slowing down</h5>
<p>I always
found that large Photoshop files would get very unwieldy, slow to open, slow to
save, slow to move groups of layers around...just about slow everything. That
doesn’t seem to happen with Sketch: the mockup of the homepage with the
illustrations of all the buildings is only 5MB and dragging or resizing groups
of layers around is never a problem.</p>
<h5>Linked layer styles</h5>
<p>Sure in Photoshop you can lock files together and then
“Select locked layers” and paste your layer styles, but it’s fiddly. Sketch
handles this in a very elegant way and layer styles on linked layers are always
kept in synch.</p>
<h5>Everything is a vector</h5>
<p>It makes sense that everything except a bitmap image
should be a vector when you’re designing mockups. It means you can easily export
a larger version of your artwork for retina screens, zooming in always looks
great and if you decide to change the size of your design later it’s not an
issue.</p>
<p>I could keep going. There are so many features that make me prefer it over
Photoshop for UI work, like the way it handles exporting assets, the built in
grid tool, artboards, its superior text rendering and the infinite canvas to
name a few. It is worth mentioning that Sketch is not a bitmap editor and
doesn’t try to be, for that you will probably still need to rely on Photoshop
or some equivalent.</p>
<h4>We built our own Compass framework</h4>
<p>I’ve used Compass for a while now but this was my first experience building and
using a proprietary Compass framework and so far it seems an ideal way to share
styles among our different websites. You have to think a bit harder about how to
break up your styles and make sure that all your common styles end up in the gem
and not mixed in with the site specific styles. This lead me to create a
guideline on how our stylesheets should be organised across all our sites and
the framework. Check it out and maybe you’ll find it useful for your own
projects:</p>
<pre><code>scss/
|
|-- utility/
| |-- _mixins.scss
| |-- _placeholders.scss
| |-- _variables.scss
| |-- ...
|
|-- base/ # Generic styles not attached to specific pages or components
| |-- _fonts.scss
| |-- _icons.scss
| |-- _reset.scss
| |-- _type.scss
| |-- ...
|
|-- layout/ # Generic styles that give structure to a page
| |-- _grid.scss
| |-- _footer.scss
| |-- _header.scss
| |-- ...
|
|-- components/
| |-- _buttons.scss
| |-- forms/ # Complex components could be split up into multiple files
| | |--_base.scss
| | |--_inline.scss
| | |--_search.scss
| |-- _subheader-nav.scss
| |-- _tables.scss
| |-- ...
|
|-- views/ # Page level css
| |-- _home.scss
| |-- _tour.scss
| |-- _about.scss
| |-- …
</code></pre>
<p>We turned our Compass framework into a Ruby gem and keep it under version
control at GitHub but if you want to create your own framework for personal
projects and you don’t need to distribute it among a team it’s a very simple
thing to set up that could save you a huge amount of time in the long run.
Here’s a
<a href="http://chriseppstein.github.io/blog/2010/08/01/building-a-personal-framework/">
good intro video</a> from Chris Epstein (creator of Compass) to get you started
and if you decide to go down the route of turning it into a distributed gem then
be sure to take a look at the
<a href="http://compass-style.org/help/tutorials/extensions/">
guidelines for Compass extensions</a>.</p>
<h4>I started using meaningful breakpoint names</h4>
<p>The site redesign involved quite a few breakpoints: horizontal breakpoints,
vertical breakpoints and pairs of breakpoints. Keeping track of them all and
what they meant was getting pretty tricky so I tried out the
<a href="https://github.com/Team-Sass/breakpoint">breakpoint gem</a> from Team
SASS and it helped keep things sane. Some of the breakpoints are still very
general eg.</p>
<pre><code><a href='https://github.com/include' class='user-mention'>@include</a> respond-to(‘between narrow and widest’) { ... }
</code></pre>
<p>Which you could also write like:</p>
<pre><code><a href='https://github.com/include' class='user-mention'>@include</a> respond-to(‘portrait tablet’) { … }
</code></pre>
<p>Other breakpoints were very specific and I could tell at a glance what they were
for, this is where the breakpoint gem really comes into its own, eg.</p>
<pre><code><a href='https://github.com/include' class='user-mention'>@include</a> respond-to(‘compact pricing table’) { … }
</code></pre>
<p>This was a tremendously enjoyable project for me to work on mainly because it
involved such a wide range of skill-sets. My time was spent between creating
illustrations in Sketch, preparing mockups of the main pages, sitting in on
brainstorming sessions for the written copy, organising styles into a reusable
framework, writing the front-end code and learning how to integrate it with Ruby
on Rails...all the while fighting with Git and the command line as only a
designer knows how.</p>
DanWe recently redesigned the WegoWise website and whilst doing so I approachedsome common problems a bit differently than I have in the past. This article isabout how I tried out some new tools and techniques to create mockups, organisemy code and write breakpoints (for responsive web design).Customizing Google Chrome - Part 2 of 32014-02-06T11:30:00-05:002014-02-06T11:30:00-05:00http://wegowise.github.io/blog/2014/02/06/customizing-google-chrome-part-2-of-3<p><strong>This is part 2 of a series on creating a Chrome extension for use by members of a development team. We use a version of this tool here at WegoWise.</strong></p>
<p>Have you ever been investigating an issue, flipping back and forth between production, staging, and your local machine, when it hits you: <em>Did I just make a change on production!?</em></p>
<p>The goal of this extension is to read the url of the tab and display an icon if it's one of the domains we're concerned with. The icon's color and text will be set according to the domain, and we will change that color and text by creating a canvas object and rendering it.</p>
<p>{% img center /images/chrome_page_action.png 390 202 'Page Action' %}</p>
<p>Ideally we would have liked to do something really flashy like change the background color of the page or perhaps change the look and feel of the browser itself. Chrome has a theming mechanism that can change how the browser looks, however at this time you cannot change the theme from an extension. You can modify the HTML of any page from an extension, but that creates potential problems because you could inadvertently change the behavior of the page. A third option is to add an icon to the location bar which is what I decided to do.</p>
<p><strong><a id='manifest'>Manifest</a></strong></p>
<p>The first thing we need to do is make a small change to the <code>manifest.json</code>. Since we need a canvas element to create our button we change background to reference an HTML file. The HTML file takes care of loading our script from the first part of the series.
<code>json
"background": {
"page": "wego_menus.html"
},
"page_action" :
{
"default_title" : "Wegotools"
}
</code>
The second change is to tell Chrome that we will be creating a page action. The only parameter required is the title of our extension. Now we have a page action which we can reference in our code.</p>
<p><strong><a id='html'>HTML</a></strong></p>
<p>For this project we're only going to use the HTML file in order to load our javascript and provide a canvas element. Chrome allows you to do a lot more with this file for example it can be rendered as a popup with html, buttons, form elements, etc. You're only limited by your imagination and the restrictions of the Chrome sandbox.
<code>html
<html>
<body>
<canvas id='canvas' width='19' height='19' style='display: none'></canvas>
<script src='wego_menus.js'></script>
</body>
</html>
</code></p>
<p><strong><a id='javascript'>Javascript</a></strong></p>
<p>For configuration information I've modified the menu object from the previous installment:
<code>javascript
var menus = { 'staging': { 'url': 'https://staging.example.com',
'color': 'orange',
'title': 'Staging' },
'localhost': { 'url': 'http://localhost:3000',
'color': 'green',
'title': 'Localhost' },
'production': { 'url': 'https://www.example.com',
'color': 'red',
'title': 'Production' } }
</code>
We need to configure Chrome to execute a function for two events, when the location bar is changed:
```javascript
chrome.tabs.onUpdated.addListener(onTabUpdated);</p>
<p>function onTabUpdated(tabId, changeInfo, tab) {
checkForValidUrl(tab);
}
<code>
and when a different tab is selected:
</code>javascript
chrome.tabs.onActivated.addListener(onTabActivated);</p>
<p>function onTabActivated(activeInfo) {
chrome.tabs.get(activeInfo.tabId, function(tab) {
if (tab && tab.url) {
checkForValidUrl(tab);
}
});
}
```
Due to the way that Chrome events fire, tab activation does not provide the currently selected tab. Instead it needs to be retrieved separately. Either way, once we've got a handle to the tab, we need to check to see if the domain for the tab url is handled by our menus:</p>
<pre><code class="javascript">function checkForValidUrl(tab) {
var wegoMenu = getWegoMenu(tab.url);
if (wegoMenu) {
showIcon(menu.title, menu.color, tab.id);
} else {
chrome.pageAction.hide(tab.id);
}
};
function getWegoMenu(url) {
if (url) {
for (var i in menus) {
menu = menus[i];
if (menu && url.indexOf(menu.url) == 0) {
return menu;
}
}
}
return null;
}
</code></pre>
<p>If we don't have one of our specified urls then we will hide the page action for this tab. Assuming that the url is for one of our domains the next step is to show the icon:
<code>javascript
function showIcon(text, color, tabId) {
chrome.pageAction.setIcon({ imageData: draw(text, color), tabId: tabId });
chrome.pageAction.setTitle({ tabId: tabId, title: text });
chrome.pageAction.show(tabId);
}
</code>
You will notice that we call the <code>draw</code> function to provide image data to the the page action. The example including the draw function can be found <a href="https://github.com/wegowise/chrome-extension-blog">in this repository</a>, which you're welcome to fork and modify as you like. You can create any type of image that you can make on a normal canvas including animations. Just remember the final image can only be 19x19 and animations are annoying. Alternatively you could just provide an appropriately size image and pass that along to the <code>setIcon</code> function.</p>
<p><strong><a id='manifest'>Next Steps</a></strong></p>
<p>The example project can be found in the <a href="https://github.com/wegowise/chrome-extension-blog">Chrome Extension Blog</a> repository.</p>
<p>The last part of this series will cover different deployment methods including packaging, signing, and distributing your extension. Depending on community interest I may also do a supplement on using an Arduino to control an RGB LED strip as part of the page action outlined in this article.</p>
JeffThis is part 2 of a series on creating a Chrome extension for use by members of a development team. We use a version of this tool here at WegoWise.Customizing Google Chrome - Part 1 of 32014-01-21T18:33:00-05:002014-01-21T18:33:00-05:00http://wegowise.github.io/blog/2014/01/21/customizing-google-chrome<p><strong>This is part 1 of a series on creating a Chrome extension for use by members of a development team. We use a version of this tool here at WegoWise.</strong></p>
<p>As a developer I often need to switch between multiple environments with the same url. This usually involves cutting and pasting a url into a new tab then changing the domain. Knowing that Google Chrome allows you to easily write extensions that can interact with web pages I decided to see if I could make my workflow a little bit easier. The goal of this minor project was to be able to change just the domain part of a url from a right click in a Chrome browser.</p>
<p>{% img center /images/chrome_custom_menu.png 372 420 "A custom menu in chrome" %}</p>
<p>The first step is to turn on developer mode on the chrome://extensions page in the upper right hand corner.</p>
<p>{% img center /images/chrome_extensions_url.png 639 133 "Chrome extensions url" %}</p>
<p>An extension is defined by a file called <code>manifest.json</code> which should be created in the root directory of your chrome extension project. The manifest defines the properties and permissions of your chrome extension.</p>
<pre><code>{
"name": "Wegotools",
"description": "Loads different servers",
"version": "0.6",
"permissions": ["contextMenus", "tabs"],
"icons": {
"16": "panda-with-glasses-small.png",
"128": "panda-with-glasses.png"
},
"background": {
"scripts": ["wego_menus.js"]
},
"manifest_version": 2
}
</code></pre>
<p>Most of the parameters are self explanatory, however there are a few items that are of particular interest.</p>
<p><strong><a href="http://developer.chrome.com/extensions/permissions.html">Permissions</a></strong></p>
<p>In our case we’re going to be adding right-click menus (i.e. <code>contextMenus</code>) and we are going to be manipulating tabs. Chrome extensions can have very intimate knowledge of your browsing history and general use of the application, so Google (and I) feel it’s important that the permissions are spelled out.</p>
<p><strong>Icons</strong></p>
<p>We’re only going to use two icons in our application, a 16x16 icon that will appear in our right-click menu and a 128x128 icon that will be displayed on the extension.</p>
<div style='diplay: table'>
<div style='display:table-cell; width: 400px; vertical-align: middle'>{% img center /images/panda-with-glasses-small.png 32 32 "Small icon" %} </div>
<div style='display:table-cell; width: 400px; vertical-align: middle'>{% img center /images/panda-with-glasses.png 128 128 "Large icon" %}</div>
</div>
<div style='clear:both'></div>
<p><strong>Background</strong></p>
<p>We specify a script file that will be loaded as a background process, which I'll cover in a little bit.</p>
<p><strong>Creating the context menus</strong></p>
<p>A context menu is created by using the <a href="http://developer.chrome.com/extensions/api_index.html">Chrome API.</a></p>
<pre><code>chrome.contextMenus.create({ "id": "localhost",
"title": "Localhost",
"onclick": genericOnClick });
</code></pre>
<p>Context menus are added in the order in which they're called in your javascript. The <code>id</code> identifies which <code>contextMenu</code> has been clicked. It's important to set this if you need to have custom behavior for a particular menu as Chrome will set a random id if one is not specified. <code>title</code> is simply the words that will appear in your <code>contextMenu</code>. The <code>onclick</code> parameter is the function that will be called when the menu is accessed, which you will note is <strong>not</strong> in quotes since it's referencing the function itself.</p>
<p>You can also add radio buttons or checkboxes to the menu. In our case we're going to specify whether the existing tab's url should be changed by setting a variable, but first let's put in a separator:</p>
<pre><code>chrome.contextMenus.create({ "type": "separator" });
</code></pre>
<p>And finally the checkbox:</p>
<pre><code>chrome.contextMenus.create({ "title": "Load in separate Tab",
"id": "loadInSeparateTab",
"type": "checkbox",
"checked": true,
"onclick": checkboxOnClick });
</code></pre>
<p>The event functions go into the .js file as well. Our example code for setting the checkbox event is as follows:</p>
<pre><code>var loadInSeparateTab = true;
function checkboxOnClick(info, tab) {
loadInSeparateTab = info.checked
}
</code></pre>
<p>You'll notice that we will use the built-in <code>checked</code> property for the checkbox and have access to both the <code>contextMenu</code> and the currently displayed <code>tab</code>. If this were a production app you would want to create properly encapsulated javascript objects, however in our simplified case we'll just set a global variable.</p>
<p>The <code>onclick</code> handler for the context menus takes the same arguments as the checkbox function. We'll use the <code>id</code> that we assigned to map into a javascript object that contains our domain information.</p>
<pre><code>var menus = { 'staging': 'https://staging.yourapp.com',
'localhost': 'http://localhost:3000',
'production': 'https://production.yourapp.com' }
function genericOnClick(info, tab) {
var url = '';
url = tab.url.split('/')
url.splice(0, 3);
url = menus[info.menuItemId] + "/" + url.join('/');
if (loadInSeparateTab) {
chrome.tabs.create({ url: url, index: tab.index+1 });
} else {
chrome.tabs.update(tab.id, { url: url })
}
}
</code></pre>
<p>The <code>loadInSeparateTab</code> variable is used to determine whether we should open a new tab or use the existing tab. The domain part of the url is pulled from the <code>menus</code> object by the tab <code>id</code>.</p>
<p><strong>Loading the extension</strong></p>
<p>To deploy our code choose 'Load unpacked extension...' on the extensions page and select the directory that contains the extension code. If everything is correct when you right-click in your browser you should see 'Wegotools' and below that choices for Staging, Localhost, and Production as well as the "Load in separate tab" checkbox. Regardless of success or failure you'll see an entry for your extension as long as your manifest file is correct.</p>
<p>{% img center /images/chrome_loaded_extension.png 741 155 'The loaded extension' %}</p>
<p>If you need want to inspect your running extension click on 'background page' which will open up a javascript console that displays any errors or console messages just like the standard Chrome developer tools.</p>
<p><strong>Next Steps</strong></p>
<p>The next part of this series will cover the creation of a <code>pageAction</code> which will show an icon in the location bar when we're on one of our domains.</p>
<p>{% img center /images/chrome_page_action.png 390 202 'Page Action' %}</p>
<p>The final installment will cover deployment of our extension.</p>
This is part 1 of a series on creating a Chrome extension for use by members of a development team. We use a version of this tool here at WegoWise.Sort Like a Human Boss2013-10-27T00:00:00-04:002013-10-27T00:00:00-04:00http://wegowise.github.io/blog/2013/10/27/sort_like_a_human_boss<p>This post will discuss the ruby gem
<a href="https://github.com/wegowise/sort_authority">sort_authority</a> and the problem it
aims to solve, which is how to quickly sort mixed number and letter strings in
Ruby in a "human" way.</p>
<p>The problem is easiest to explain with images.</p>
<p>Which would you rather see in an application?</p>
<div>
<div style='width: 400px; float:left; vertical-align: center'>This? {% img center /images/bad-sorting.png /images/bad-sorting.png 156 298 "Wagh!! So bad!" %} </div>
<div style='width: 400px; float:left'>or this? {% img center /images/good-sorting.png /images/good-sorting.png 160 324 "Ah! Meaning is restored!" %}</div>
</div>
<div style='clear:both'></div>
<p>As Jeff Atwood noted <a href="http://www.codinghorror.com/blog/2007/12/sorting-for-humans-natural-sort-order.html">in this blog post</a>
from way back in 2007, this is the difference between <strong>"ASCIIbetical"</strong> and
<strong>alphabetical</strong> sorting. When creating an interface for humans with listings
containing a mix of letters and numbers you almost always want the latter and
not the former. Yet real alphabetization (also refered to as "natural" or
"human" sorting) is typically not included in standard libraries, Ruby's
included.</p>
<p>As the developer involved in implementing natural sorting for our app at
WegoWise I first turned to the solution presented
<a href="http://www.davekoelle.com/files/alphanum.rb">here</a>, which we'll call
<code>sensible_sort</code>. This algorithm works fine<sup id="fnref:1"><a href="#fn:1" rel="footnote">1</a></sup>, in that it does in fact sort
strings alphabetically. There are also plenty of gems we could have used as
well, including <a href="https://rubygems.org/gems/naturalsort">naturalsort</a>,
<a href="https://rubygems.org/gems/naturally">naturally</a><sup id="fnref:2"><a href="#fn:2" rel="footnote">2</a></sup>, and
<a href="https://rubygems.org/gems/natcmp">natcmp</a>.</p>
<p>Any of these approaches would have gotten the job done, but my coworker
<a href="https://github.com/fixlr">Nathan</a> took one look at the pull request and asked
me how well the algorithm <em>performed</em>. I shrugged<sup id="fnref:3"><a href="#fn:3" rel="footnote">3</a></sup>; I hadn't worried about
performance because the lists I was sorting were so small. Then I signed off
for the day.</p>
<p>By the next morning Nathan had thrown together some benchmarks that showed that
sensible sort could be fairly slow given a long enough list. The other Rubygems
didn't perform very well either, compared to Ruby's native ASCIIbetical sort
and a C algorithm by Martin Pool called
<a href="http://sourcefrog.net/projects/natsort/strnatcmp.c">natsort</a>. In fact,
Nathan had gone even further and made a wrapper for natsort using FFI,
labeled <code>strnatcmp.c</code> below. Behold the damning numbers:</p>
<pre><code>Sorting ['x1'] * 100_000:
user system total real
sort 0.000000 0.000000 0.000000 ( 0.001758)
strnatcmp.c 0.030000 0.000000 0.030000 ( 0.029083)
naturalsort gem 0.060000 0.000000 0.060000 ( 0.061879)
naturally gem 0.990000 0.020000 1.010000 ( 1.009634)
natcmp gem 1.140000 0.010000 1.150000 ( 1.141950)
sensible_sort 2.280000 0.000000 2.280000 ( 2.286274)
</code></pre>
<p>Considering that we're talking about 100,000 items maybe 2 seconds isn't that
bad. We don't have user-facing lists that have 100,000 elements... yet. Also,
I had developed a sentimental attachment to <code>sensible_sort</code> and decided to
write a more realistic benchmark involving randomized strings (<a href="https://gist.github.com/tristil/6136694">gist here</a>),
resulting in the following numbers:</p>
<pre><code>Sorting 100,000 strings like this: "9 3q1q05_xsfw87i6wf7"
user system total real
sort 0.110000 0.000000 0.110000 (0.112114)
strnatcmp.c 0.560000 0.000000 0.560000 (0.560879)
naturally gem 7.940000 0.020000 7.960000 (7.965125)
sensible_sort 21.230000 0.160000 21.390000 (21.406591)
natcmp gem 51.150000 0.190000 51.340000 (51.364688)
naturalsort gem 161.590000 1.200000 162.790000 (162.883611)
(This benchmark was made a while ago on Ruby 1.9.2 but results hold for later
Rubies)
</code></pre>
<p>I was pleased to note that <code>sensible_sort</code> is <strong>not</strong> the worst approach in
this benchmark. But it's also not very good. 21 seconds is pretty much
unacceptable for an operation you might want to do inside a page request.
Again, we don't often have to sort 100,000 items for humans these days, but
this is an extension on the core <code>Enumerable</code> module that might get used in
various contexts throughout the application. If we can avoid an inherently slow
approach in favor of a painless alternative, why wouldn't we?</p>
<p>And that's how the
<a href="https://github.com/wegowise/sort_authority">sort_authority</a><sup id="fnref:4"><a href="#fn:4" rel="footnote">4</a></sup>
gem came about. <code>sort_authority</code> is just a simple wrapper around <code>strnatcmp.c</code>,
plus an extension on <code>Enumberable</code> to add <code>#natural_sort</code> and <code>#natural_sort_by</code>
methods on arrays and such. If you care about sorting lists in a sane order
(and you should) and want it to be almost as fast as regular <code>sort</code> (why not?)
then you should give it a try.</p>
<div class="footnotes">
<hr/>
<ol>
<li id="fn:1">
<p>Although some changes involving type coercion were required to update the code for Ruby 1.9+, available <a href="https://gist.github.com/tristil/7189713">in this gist</a>).<a href="#fnref:1" rev="footnote">↩</a></p></li>
<li id="fn:2">
<p>Which has the distinction of being one of the least googleable Rubygems.<a href="#fnref:2" rev="footnote">↩</a></p></li>
<li id="fn:3">
<p>Well, our team is mostly remote so this communication took place in the form of emoticons and animated gifs in our chat room, but same idea.<a href="#fnref:3" rev="footnote">↩</a></p></li>
<li id="fn:4">
<p><a href="https://github.com/jdewyea">Jed</a> came up with the awesome name.<a href="#fnref:4" rev="footnote">↩</a></p></li>
</ol>
</div>
JoeThis post will discuss the ruby gemsort_authority and the problem itaims to solve, which is how to quickly sort mixed number and letter strings inRuby in a "human" way.Git Rebase as Time Travel2013-08-21T18:50:00-04:002013-08-21T18:50:00-04:00http://wegowise.github.io/blog/2013/08/21/git-rebase-as-time-travel<p>At WegoWise we use extensive code review feedback to iteratively improve code.
I want to describe how Git fits into this process, because this is probably the
biggest change I had to make to my preexisting workflow when I came on board.
Basically I had to relearn how to use Git. The new way of using it (that is, it
was new to me) is extremely powerful and in a strange way extremely satisfying,
but it does take a while to get used to.</p>
<h1>Importance of rebasing</h1>
<p>I would describe my old approach and understanding as ”subversion, but with
better merging” <sup id="fnref:1"><a href="#fn:1" rel="footnote">1</a></sup> . I was also aware of the concept of rebasing from having
submitted a pull request to an open source project at one point, but I didn’t
use it very often for reasons I’ll discuss later. As it turns out understanding
<code>git rebase</code> is the key to learning how to use Git as more than a ‘better
subversion’.</p>
<p>For those who aren’t familiar with this command, <code>git rebase <branch></code> takes
the commits that are unique to your branch and places them “on top” of another
branch. You typically want to do this with <code>master</code>, so that all your commits
for your feature branch will appear together as the most recent commits when
the feature branch is merged into <code>master</code>.</p>
<p>Here’s a short demonstration. Let’s say this is your feature branch, which
you’ve been developing while other unrelated commits are being added to master:</p>
<p>{% img center /images/Screen-Shot-2013-03-16-at-11.57.27-PM-e1363492819169.png /images/Screen-Shot-2013-03-16-at-11.57.27-PM-e1363492819169.png 507 105 "Feature branch with ongoing mainline activity" %}</p>
<p>If you merge without rebasing you’ll end up with a history like this:</p>
<p>{% img center /images/Screen-Shot-2013-03-17-at-12.04.20-AM-e1363493583521.png /images/Screen-Shot-2013-03-17-at-12.04.20-AM-e1363493583521.png 287 560 "History is all jacked up!" %}</p>
<p>Here is the process with rebasing:</p>
<pre><code class="bash"># We're on `feature_branch`
git rebase master # Put feature_branch's commits 'on top of' master's
git checkout master
git merge feature_branch
</code></pre>
<p>This results in a clean history:</p>
<p>{% img center /images/Screen-Shot-2013-03-17-at-12.35.07-AM-e1363495203367.png /images/Screen-Shot-2013-03-17-at-12.35.07-AM-e1363495203367.png 289 479 "Feature branch commits on top" %}</p>
<p>Another benefit of having done a rebase before merging is that there’s no need
for an explicit merge commit like you see at the top of the original history.
This is because — and this is a key insight — the feature branch
is exactly like the master branch but with more commits added on. In other
words, when you merge it’s as though you had never branched in the first place.
Because Git doesn’t have to ‘think’ about what it’s doing when it merges a
rebased branch it performs what is called a <strong>fast forward</strong>. In this case
it moved the HEAD from <code>899bdb</code><sup id="fnref:2"><a href="#fn:2" rel="footnote">2</a></sup> (More mainline activity) to <code>5b475e</code>
(Finished feature branch).</p>
<p>The above is the basic use case for git rebase. It’s a nice feature that keeps
your commit history clean. The greater significance of git rebase is the way it
makes you think about your commits, especially as you start to use
the interactive rebase features discussed below.</p>
<h1>Time travel</h1>
<p>When you call git rebase with the interactive flag, e.g. git rebase -i master,
git will open up a text file that you can edit to achieve certain effects:</p>
<p>{% img r /images/Screen-Shot-2013-03-09-at-6.07.05-PM.png /images/Screen-Shot-2013-03-09-at-6.07.05-PM.png 523 314 "Interactive rebase menu" %}</p>
<p>As you can see there are several options besides just performing the rebase
operation described above. Delete a line and you are telling Git to disappear
that commit from your branch’s history. Change the order of the commit lines
and you are asking Git to attempt to reorder the commits themselves. Change the
word <code>pick</code> to <code>squash</code> and Git will squash that commit together with the
commit on the preceding line. Most importantly, change the word <code>pick</code> to
<code>edit</code> and Git will drop you just after the selected ref number.</p>
<p>I think of these abilities as time travel. They enable you to go back in the
history of your branch and make code changes as well as reorganize code into
different configuration of commits.</p>
<p>Let’s say you have a branch with several commits. When you started the branch
out you thought you understood the feature well and created a bunch of code to
implement it. When you opened up the pull request the first feedback you
received was that the code should have tests, so you added another commit with
the tests. The next round of feedback suggested that the implementation could
benefit from a new requirement, so you added new code and tests in a third
commit. Finally, you received feedback about the underlying software design
that required you to create some new classes and rename some methods. So now
you have 4 commits with commit messages like this:</p>
<p>{% img r /images/Screen-Shot-2013-03-16-at-6.56.16-PM-e1363474670152.png /images/Screen-Shot-2013-03-16-at-6.56.16-PM-e1363474670152.png 280 386 "A messy commit history" %}</p>
<ol>
<li>Implemented new feature</li>
<li>Tests for new feature</li>
<li>Add requirement x to new feature</li>
<li>Changed code for new feature</li>
</ol>
<p>This history is filled with useless information. Nobody is going to care in the
future that the code had to be changed from the initial implementation in
commit 4 and it’s just noise to have a separate commit for tests in commit 2.
On the other hand it might be valuable to have a separate commit for the added
requirement.</p>
<p>To get rid of the tests commit all you have to do is squash commit 2 into
commit 1 (by entering interactive rebase and changing the word <code>pick</code> on the
second commit to <code>squash</code> or just <code>s</code><sup id="fnref:3"><a href="#fn:3" rel="footnote">3</a></sup>), resulting in:</p>
<ol>
<li>Implemented new feature</li>
<li>Add requirement x to new feature</li>
<li>Changed code for new feature</li>
</ol>
<p>New commit 3 has some code that belongs in commit 1 and some code that belongs
with commit 2. To keep things simple, the feature introduced in commit 1 was
added to <code>file1.rb</code> and the new requirement was added to <code>file2.rb</code>. To handle
this situation we’re going to have to do a little transplant surgery. First we
need to extract the part of commit 3 that belongs in commit 1. Here is how I
would do this:</p>
<pre><code class="bash"># We are on HEAD, i.e. commit 3
git reset HEAD^ file1.rb
git commit --amend
git stash
git rebase -i master
# ... select commit 1 to edit
git stash apply
git commit -a --amend
git rebase --continue
</code></pre>
<p>It’s just that easy! But seriously, let’s go through each command to understand
what’s happening.</p>
<ol>
<li>The first command, <code>git reset</code>, is notoriously hard to explain, especially
because there’s another command, <code>git checkout</code>, which seems to do something
similar. The diagram at the top of <a href="http://stackoverflow.com/questions/3639342/whats-the-difference-between-git-reset-and-git-checkout">this Stack Overflow
page</a> is
actually extremely helpful. The thing about Git to repeat like a mantra is that
Git has a two step commit process, staging file changes and then actually
committing. Basically, when you run <code>git reset REF</code> on a file it stages the
file for committing at that ref. In the case of the first command, <code>git reset
HEAD^ file.rb</code>, we’re saying “stage the file as it looked before HEAD’s
change”; in other words, revert the changes we made in the last commit.</li>
<li>The second command, <code>git commit --amend</code> commits what we’ve staged into
HEAD (commit 3). The two commands together (a reset followed by an amend) have
the effect of uncommitting the part of HEAD’s commit that changed file1.rb.</li>
<li>The changes that were made to file1.rb aren’t lost, however. They were
merely uncommitted and unstaged. They are now sitting in the working directory
as an unstaged diff, as if they’d never been part of HEAD. So just as you could
do with any diff you can use <code>git stash</code> to store away the diff.</li>
<li>Now I use interactive rebase to travel back in time to commit 1. Rebase
drops me right after commit 1 (in other words, the temporary rebase HEAD is
commit 1).</li>
<li>I use <code>git stash apply</code> to get my diff back (you might get a merge conflict
at this point depending on the code).</li>
<li>Now I add the diff back into commit 1 with <code>git commit --amend -a</code> (-a
automatically stages any modified changes, skipping the <code>git add .</code> step).</li>
</ol>
<p>This is the basic procedure for revising your git history (at least the way I
do it). There are a couple of other tricks that I’m not going to go into detail
about here, but I’ll leave some hints. Let’s say the changes for the feature
and the new requirement were both on the same file. Then you would need to use
<code>git add --patch file1.rb</code> before step 2. What if you wanted to introduce a
completely new commit after commit 1? Then you would use interactive rebase to
travel to commit 1 and then add your commits as normal, and then run <code>git
rebase --continue</code> to have the new commits inserted into the history.</p>
<h1>Caveats</h1>
<p>One of the reasons I wasn’t used to this workflow before this job was because I
thought rebasing was only useful for the narrow case of making sure that the
branch commits are grouped together after a merge to master. My understanding
was that other kinds of history revision were to be avoided because of the
problems that they cause for collaborators who pull from your public repos.
I don’t remember the specific blog post or mailing list message but I
took away the message that once you’ve pushed something to a public repo (as
opposed to what’s on your local machine) you are no longer able to touch that
history.</p>
<p>Yes and no. Rebasing and changing the history of a branch that others are
pulling from can cause a lot of problems. Basically any time you amend a commit
message, change the order of a commit or alter a commit you actually create a
new object with a new sha reference. If someone else naively pulls from your
branch after having pulled the pre-revised-history they will get a weird set of
duplicate code changes and things will get worse from there. In general if
other people are pulling from your public (remote) repository you should not
change the history out from under them without telling them. Linus’ guidelines
about rebasing
<a href="http://www.mail-archive.com/dri-devel@lists.sourceforge.net/msg39091.html">here</a> are
generally applicable.</p>
<p>On the other hand, in many Git workflows it’s not normal for other people to be
pulling from your feature branch and if they are they shouldn’t be that
surprised if the history changes. In the Github-style workflow you will
typically develop a feature branch on your personal repository and then submit
that branch as a pull request to the canonical repository. You would probably
be rebasing your branch on the canonical repository’s master anyway. In that
sense even though your branch is public it’s not really intended for public
consumption. If you have a collaborator on your branch you would just shoot
them a message when you rebase and they would do a “hard reset” on their branch
(sync their history to yours) using <code>git reset --hard
remote_repo/feature_branch</code>. In practice, in my limited experience with a
particular kind of workflow, it’s really not that big a deal.</p>
<h1>Don’t worry</h1>
<p>Some people are wary of rebase because it really does alter history. If you
remove a commit you won’t see a note in the commit history that so and so
removed that commit. The commit just disappears. Rebase seems like a really
good way to destroy yours and other people’s work. In fact you can’t actually
screw up too badly using rebase because every Git repository keeps a log of the
changes that have been made to the repository’s history called the reflog.
Using <code>git reflog</code> you can always back out misguided recent history changes by
returning to a point before you made the changes.</p>
<p>Hope this was helpful!</p>
<p><em>Note: a different version of this post was posted on the author's blog <a href="http://www.stupididea.com/2013/03/17/git-workflow/">Stupid Idea</a></em>.</p>
<div class="footnotes">
<hr/>
<ol>
<li id="fn:1">
<p>Not an insignificant improvement, since merging in Subversion sucks.<a href="#fnref:1" rev="footnote">↩</a></p></li>
<li id="fn:2">
<p>Which I always think about as a hard drive head, which I in turn think about as a record player needle.<a href="#fnref:2" rev="footnote">↩</a></p></li>
<li id="fn:3">
<p>Or use <code>f</code> for <code>fixup</code> to combine the two commits but only use the commit message from the first.<a href="#fnref:3" rev="footnote">↩</a></p></li>
</ol>
</div>
JoeAt WegoWise we use extensive code review feedback to iteratively improve code.I want to describe how Git fits into this process, because this is probably thebiggest change I had to make to my preexisting workflow when I came on board.Basically I had to relearn how to use Git. The new way of using it (that is, itwas new to me) is extremely powerful and in a strange way extremely satisfying,but it does take a while to get used to.method_name_missing2013-07-05T14:03:00-04:002013-07-05T14:03:00-04:00http://wegowise.github.io/blog/2013/07/05/method-name-missing<p>The WegoWise team spends a lot of time talking about names. We try to pick
variable, class and method names that express not only the function but also
the <em>intention</em> of code. The idea is that good names help other developers by:</p>
<ul>
<li>Helping them reason about the code at a high level. Good names combined with
short methods should approach the readability of pseudocode.</li>
<li>Reducing the need for extra comments, which tend to get out of sync with code.</li>
<li>Clarifying the purpose of an object. Sometimes if you can't find a good name
for something it's because it's trying to do too much or is otherwise
ill-conceived. Sometimes a good name will suggest other objects to create.</li>
</ul>
<p>We will often suggest alternate names to each other during code review. It is
also common for a developer to try out a couple names in the developer
chatroom before settling on one.</p>
<p>This is the tale of one such occasion. Note that this is an incredibly trivial
example, but it's meant to demonstrate the discussion process we use to arrive
at a good name.</p>
<p>I was pairing with another developer on a design that let you attach arbitrary
traits to a record. We had a bit of code that checked to make sure that a value
was present, defined as not <code>nil</code> and not an empty string. <code>0</code> was a valid
value. Active Support's <code>.blank?</code> wouldn't work because it detects <code>0</code> as well
as <code>''</code> and <code>nil</code>. In Rails this is normally less of an issue because you're
usually working with a string field <em>or</em> an integer field, but in our case we
had a string field that could be coerced to different types.</p>
<p>The code looked like this:</p>
<pre><code class="ruby">read_attribute(:value).nil? || read_attribute(:value) == ''
</code></pre>
<p>This would probably be good enough if we were using it in one place, but we
were using this logic in a couple places, so we thought we should encapsulate
it in a method. Also, even though this logic is certainly clear as to <em>what</em> it
does, it's slightly mysterious as to <em>why</em> it's doing it. Nothing terrible, but
it might cause a developer to slow down for a second and wonder, "What's so
special about <code>nil</code> and <code>''</code>? Why didn't they just use <code>.blank?</code>" A second that
could be spared with a good, explanatory name.</p>
<p>So we asked around in the chatroom. Here were a couple suggestions of greater
and lesser absurdity:</p>
<ul>
<li><code>.not_nil_or_empty_string?</code></li>
<li><code>.present_but_not_false_and_blank_but_not_empty?</code></li>
<li><code>.peanut_butter?</code></li>
<li><code>.nathans_choice?</code></li>
<li><code>.has_value?</code></li>
</ul>
<p>I think it's safe to filter the list of candidates down to the first and last
options, <code>.not_nil_or_empty_string?</code> and <code>.has_value?</code>. The first one has the
virtue of describing exactly what's going on, but of course it's rather silly,
since the method name is almost the same as the expression it contains. The
second one is better but it reverses the question and clashes with a method on
<code>Hash</code> that takes an argument.</p>
<p>At this point one of the developers (the same one who suggested
<code>.peanut_butter?</code> actually) suggested <code>.missing_value?</code>. He pointed out that
this method name expressed the <em>intention</em> of the code. Specifically we were
asking, "Does the field not have a usable value? Is it missing a value?" This
name exposes an additional flaw in the silly name above: that name told the
developer <em>too much</em> about the implementation of the method. Another developer
should only have to know about the criteria used to determine validity if it
directly concerns the problem they're working on. For most cases it's enough to
know that the field validates against "missing" values.</p>
<p>As noted above, this is a trivial example. It may seem like it was a waste of
time to discuss this as a group, but in all it took about 10 minutes to work
through the options. Because finding good names is part of our discipline as a
group, our codebase as a whole is easier to understand and reason about than it
would be otherwise.</p>
JoeThe WegoWise team spends a lot of time talking about names. We try to pickvariable, class and method names that express not only the function but alsothe intention of code. The idea is that good names help other developers by:Rails Gotcha: Saving STI Records2013-05-09T18:34:00-04:002013-05-09T18:34:00-04:00http://wegowise.github.io/blog/2013/05/09/rails-gotcha-saving-sti-records<p>Let's say you have a persisted instance of a model, <code>RedCar</code>, that uses STI and
is descended from <code>Car</code>. You have an action that changes the type of the record
to <code>BlueCar</code>. You use <code>#becomes</code> to cast the record to the new type <em>before</em>
the save because you want to use the validations and callbacks from the new
model:</p>
<pre><code class="ruby">car = Car.find(1)
car.type # => 'RedCar'
car = car.becomes(BlueCar)
car.type # => 'BlueCar'
car.save!
</code></pre>
<p>But wait! If you retrieve <code>car</code> from the database again you'll see that it's
still an instance of <code>RedCar</code>. In fact, you'll see that no changes are saved to
the database:</p>
<pre><code class="ruby">car = Car.find(1)
car.brand # => 'GM'
car = car.becomes(BlueCar)
car.brand = 'Ford'
car.save!
car = Car.find(1)
car.type # => 'RedCar'
car.brand # => 'GM'
</code></pre>
<p>Doing an explicit <code>update_column</code> on <code>type</code> here doesn't help:</p>
<pre><code class="ruby">car = Car.find(1)
car = car.becomes(BlueCar)
car.update_column(:type, 'BlueCar')
car = Car.find(1)
car.type # => 'RedCar'
</code></pre>
<p>Also, note: no exceptions are raised. We'll save you any more head-scratching
by pointing out the SQL that is generated:</p>
<pre><code class="sql">UPDATE `cars` SET `type` = 'BlueCar', `brand` = 'Ford'
WHERE `cars`.`type` IN ('BlueCar') AND `cars`.`id` = 1
</code></pre>
<p>The problem is that Rails STI scopes the statement to <code>WHERE `cars`.`type` IN
('BlueCar')</code> even for UPDATES and even when you're trying to change the type
itself. Woops.</p>
<p>So what's the solution? As far as we can tell you have to do a separate update
after you've passed validation. We ended up implementing this in a
before_filter along these lines:</p>
<pre><code class="ruby">class Car < ActiveRecord::Base
before_update :really_change_type, :if => :type_changed?
def really_change_type
Car.where(id: id).update_all(type: type)
end
end
</code></pre>
JoeLet's say you have a persisted instance of a model, RedCar, that uses STI andis descended from Car. You have an action that changes the type of the recordto BlueCar. You use #becomes to cast the record to the new type beforethe save because you want to use the validations and callbacks from the newmodel: