This is not a project I wanted to do, but I couldn’t find a library that did everything I needed it to. One library had support for creating shipments, but didn’t have support for getting a list of rates. Another had support for getting rates, but didn’t have support for creating shipments. I need both for the application I’m working on. I even decided to modify one of the libraries to suit my needs but the code was really hard to follow and I didn’t feel comfortable using it. Especially since shipping was such an important part of my application.

That’s when I came to the sad realization that I probably need to write my own library. I was really surprised by the lack of choices when it came to shipping API integration. Then I started reading the FedEx documentation and realized why. Documentation on creating a shipment with the FedEx web services was 570 pages long. The amount of bullshit in the documentation astounded me. 99% of developers are probably concerned with maybe 30 pages in that entire document. I felt like I was reading a legal document, not web service documentation.

Sorry for the ranting, but this week has been especially frustrating, imagine reading FedEx documentation all day. Anyways, onto Shippinglogic.

What is Shippinglogic?

My goal with Shippinglogic was to make integrating with the various shipping APIs (FedEx, UPS, USPS, etc) easy and complete. The keyword being complete. Shippinglogic is not an abstraction layer, meaning each classes’ focus is on the web service at hand. It is not trying to conform to an interface. With all of the differences between the various APIs its simply not possible to do this without severely limiting the feature sets for each API. There is no defined shipping standard that all of these companies agreed on, so how do you safely make an abstraction layer for this?

Anyways, I think it turned out pretty well. The syntax is pretty slick, thanks to the proxy class trick I talked about in a previous post.

Here’s a quick example of tracking a package:

fedex = Shippinglogic::FedEx.new(key, password, account, meter)
tracking = fedex.track(:tracking_number => "XXXXXXXXXXXXXXXXX")

# this shows that the tracking object is actually a proxy for the underlying array
tracking.tracking_number
# => "XXXXXXXXXXXXXXXXX"

tracking.tracking_number = "YYYYYYYYYYYYYYY"
# => "YYYYYYYYYYYYYYY"

tracking.class
# => Array

tracking.first
# => #

tracking.first.name
# => "Delivered"

I was pretty thorough with the documentation. Check out the project on github for more examples. Let me know what you think.

user = User.first
user.orders.build
# => #Order object

user.orders.first
# => #Order ojbect

user.orders.class
# => Array

Wait, ‘build’ isn’t a method available for arrays. How did they do that?

One way is to add a ‘build’ method into the Array class, but this is really intrusive. You don’t necessarily need a build method for every Array object, more importantly build needs to know what kind of object to build. How does it get this information?

The answer to the mystery is a proxy class that delegates method calls to the underlying array. It’s actually really simple to do. Check it out:

class Proxy
    instance_methods.each { |m| undef_method m unless m =~ /(^__|^send$|^object_id$)/ }

    def my_awesome_method
       "you just called an awesome method!"
    end

    protected
        def method_missing(name, *args, &block)
          target.send(name, *args, &block)
        end

        def target
          @target ||= []
        end
end

Notice that the first thing we do is undefine just about every method in the class. This makes sure that calls to those methods hit method_missing and ultimately get called on the underlying array. Afterwards we define any special methods, such as ‘my_awesome_method’, and then we handle any calls to undefined methods. That’s it! It’s actually really simple and clever. Here’s how it works:

proxy = Proxy.new
proxy.my_awesome_method
# => "you just called an awesome method!"

proxy.class
# => Array

Why would you want to do this?

Well sometimes you want to add a little extra functionality to a basic object, such as an Array, Hash, etc. This allows you to do that without having to modify the underlying class, which is good. You basically get to play middle man and intercept any calls to that object. Then you have the decision to pass them through or do your own thing. This also promotes much more “ruby like” syntax. Without this, calling associations in AR would look something like:

user.orders.build
user.orders.create
user.orders.all.first
user.orders.all.size

Anything related to the underlying Array must be called off of the “all” method. That just seems dirty. This is so much cleaner:

user.orders.build
user.orders.create
user.orders.first
user.orders.size

Pretty cool huh?

I’m not going to ramble about the changes, because you can read my previous post or you can check out the read me on github.

Using github not rubyforge

I have decided to start using github to host my gems, if you want to start using v2 as a gem you need to install the binarylogic-searchlogic gem from the github source. (checkout the read me for easy installation instructions).

Thank you

Sorry for such a quick post. I figured this was an important announcement.

Thank you for being patient with v1, I know there were some people having issues. This should address all of them.

As always, I did my best with the readme and the rdocumentation. If you really want to see what v2 is all about I suggest you take a glance at those, they should be comprehensive.

I hope you enjoy v2, I’m really excited to release it. I have already integrated it into the app that I am working on, and I love it. It works great and all of my tests pass.

So, the project I’m working on now required something from Searchlogic v1 that it couldn’t provide. That is, searching with existing named scopes. I could do this, but it wasn’t as elegant as it was with the built in conditions (Ex: username_like, etc.).

The idea behind Searchlogic v2

So this got me thinking, how can I get Searchlogic v1 to use existing named scopes? That’s when it hit me, why not use named scopes for everything?I think we can all agree named scopes are the heart of searching and breaking up common search logic in your models. Instead of having all of this crazy logic write SQL and chain it together, as in v1, why not have Searchlogic dynamically create these common named scopes for you and then just leverage those? That’s exactly what v2 does. Instead of going into detail about v2, check out the current README. It explains everything v2 has to offer up to this point.

The best part about this approach is that it fits nicely into your application. All that it is doing is creating named scopes. More importantly, if you need some condition that Searchlogic doesn’t offer, no problem, just create your own named scope. This is what you should be doing anyways.

To bring it all home, let me show a simple search form that uses v2. Assume we have

User(name:string, age:integer)
Order(user_id:integer, total:float)

User.has_many :orders
User.named_scope :four_year_olds, :conditions => {:age => 4}
User.named_scope :name_sounds_like, lamda { |value| {:conditions => ["name SOUNDS LIKE ?", value]} }

Your search form could look like this:

- form_for @search do |f|
  = f.text_field :name_like
  = f.select :age_greater_than, (1..100)
  = f.text_field :orders_total_less_than
  = f.text_field :name_sounds_like
  = f.check_box :four_year_olds
  = f.submit

Notice the use of custom named scopes. Also the other conditions are also named scopes that searchlogic dynamically defines as you need them.

The best part of v2

• V1: ~2300 lines of code
• V2: ~350 lines of code

As you might have saw in this post Searchlogic v1 was too big for me to handle. The issues people were having would take way too long to solve and I just didn’t like the code. The library really wasn’t advancing and it was just an annoyance. It was regretably the first library I ever wrote, and as such there are a lot of things I would do differently, which should be evident in v2.

That being said, v2 is lean, makes sense, and just feels right. I’m really looking forward to maintaing the library and seeing where it goes. I hope everyone enjoys it. Let me know what you think. There are just a couple of things left to do before I officially release it.

Applications using v1

Lastly, because this is such a big change, backwards compatibility will be broken. Before I officially release the library I will write a document explaining the differences and how to transition. In the mean time, your gem declarations in your application should look like this, so that your app doesn’t break when the new version is released:

 config.gem "searchlogic", :version => "~> 1.6.6"
# the ~ will let you advance minor and tiny versions, but not major versions.

I think this is just the beginning though, here are my thoughts….

The Big Picture

Thanks to things like Rack, the MVC design pattern, and ActiveRecord / DataMapper, all of these frameworks look very similar. They all have a similar pattern, which is the inherent nature of the MVC design pattern. So I decided to take advantage of this with Authlogic. Any specific framework implementation is extracted out into an adapter. Similar to how ActiveRecord has an adapter for each database type, except the Authlogic adapters are much simpler.

Probably the main reason these adapters are so simple / similar is because of Rack. Thanks to Rack, there is a request standard across all of these frameworks. “But Ben, why not skip all of this nonsense and implement this right into Rack?”. The reason I didn’t go right into Rack is…

1. Not every “Rack compatible” framework uses the Rack libraries. After digging through the rails internals I found that the RackRequest class extends AbstractRequest, which has nothing to do with Rack. Lastly, older versions of rails aren’t rack compatible.
2. I need to hook into the controller anyways with a before_filter
3. With my implementation you get the best of both worlds. I designed the AbstractAdapter to conform to the rack standards, and any framework specific implementation can subclass this and do it’s magic.

So what’s the big picture here? The big picture is that there is no reason Authlogic can’t be an authentication solution for virtually any ruby framework out there. Why reinvent the wheel for every framework?

What about framework X?

Since writing adapters for rails and merb was so simple, why not keep going? Here are a few frameworks I plan to write an adapter for:

1. Mack
2. Ramaze
3. Sinatra

If you are feeling generous and want to write an adapter for a framework other than rails or merb, please do so. Let me know about it and I will add it into the source.

Final thoughts

Authlogic loves you regardless of your framework

Why modifiers are necessary?

ActiveRecord does a great job when it comes to keeping your code database agnostic. But I feel like it neglected searching when it came to that goal. What if you want to find all records that were created after 7am? Depending on your database you would have to do something like the following:

MySQL:        HOUR(created_at)
PostgreSQL:   date_part('hour', created_at)
SQLite:       strftime('%H', created_at)

All of a sudden your app is not database agnostic. Oh no!

All of that bragging to your girlfriend about how you can switch databases in a matter of minutes was all a lie. How’s she going to feel when she finds out HOUR(created_at) is a MySQL specific function? Searchlogic to the rescue!

Searchlogic modifiers

Remember how you can do this in searchlogic?

User.all(:condition => {:created_at_gt => Time.parse("Jan 1, 2008")})
# => "created_at > 2008-01-01 00:00:00

Now you can do this too:

User.all(:conditions => {:hour_of_created_at_gt => 10})
# => HOUR(created_at) > 10

or this:

User.all(:conditions => {:tan_of_sin_of_cos_of_minute_of_created_at_gt => 20})
# => TAN(SIN(COS(MINUTE(created_at)))) > 20

What the hell was that? That was me making the point that you can chain modifiers. Obviously a very unrealistic example, but cool none-the-less.

Check out the readme for a list of modifiers and check out the changelog for a complete list of changes

What do you think?

Maybe it’s not a pain in the ass for you, maybe it is, or maybe it is and you don’t know it. We’ll never know. Either way, this tutorial is pain-in-the-ass free. Guaranteed. That’s what this is all about, making your life easier when it comes to paginating, ordering, and searching. Things you do multiple times in almost every application.

My solution: Searchlogic. It has saved me a lot time, shortened my code, and ultimately given me the proper tools to paginate, order, and search my data. Hopefully it will do the same for you.

Enough talk, let’s dive in.

“Does this really work? Am I about to waste my time?”. I hope not. But just for you, I made this live example:

Live example based on this tutorial

Paginating, ordering, and searching doesn’t get any easier than this. Before we start I want to make the following assumptions about you:

1. You have a working rails application on rails edge (rake rails:freeze:edge)
2. You have the following model structure: UserGroup => User => Order (=> = has_many). Make sure you have the has_many and belongs_to relationships set up. The fields are irrelevant, just change the field names in the view to whatever you have.

1. Install Searchlogic

$ sudo gem install searchlogic

Now add the gem dependency to your rails config:

# config/environment.rb
config.gem "searchlogic"

As always you can install searchlogic as a plugin, but plugins are becoming a thing of the past. I recommend the above approach, but here is the plugin installation if you prefer to do it this way:

$ script/plugin install git://github.com/binarylogic/searchlogic.git

2. Create your controller

Let’s create a controller that searches, orders, and paginates users. Like an admin area. Create app/controllers/users_controller.rb with the following content:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
    def index
        @search = User.new_search(params[:search])
        @users, @users_count = @search.all, @search.count
    end
end

What did we do here?

It’s simple actually, if you haven’t looked at Searchlogic I suggest you take a quick glance. I’m not going to get into crazy detail about what it does, because the README in the library covers that, but to make this short it “enhances” searching with ActiveRecord.

Notice that we started a new search with the @search object. This lets you search via an object, which is really handy for your view, which you will see below. It also added in some nifty methods such as page, per_page, order_by, order_as, and a plethora of conditions for each of your columns. The second line does a simple search and counts the results. The thing to remember here is that the .count method ignores pagination. It ignores the limit and offset values. So if you want to know how many users matched the search, use count, not @users.size. If the first page is limited to 10 reconds @user.size will return 10.

3. Create the view and your’re done

# app/views/users/index.html.erb
<% if @users_count > 0 %>
    <%= @users_count %> users found

    <table border="1" cellpadding="5">
        <tr>
            <th><%= order_by_link :id %></th>
            <th><%= order_by_link :user_group => :name %></th>
            <th><%= order_by_link :first_name %></th>
            <th><%= order_by_link :last_name %></th>
            <th><%= order_by_link [:email, :first_name] %></th>
        </tr>
        <% @users.each do |user| %>
            <tr>
                <td><%= user.id %></td>
                <td><%= user.user_group ? user.user_group.name : "-" %></td>
                <td><%= user.first_name %></td>
                <td><%= user.last_name %></td>
                <td><%= user.email %></td>
            </tr>
        <% end %>
    </table>

    <br />
    <br />

    Per page: <%= per_page_select %>

    <% if @search.page_count > 1 %>
        <br />Page: <%= page_select %>
    <% end %>
<% else %>
    No users were returned
<% end %>

What did we do here?

All that we did was iterate through our users, list them in a table format, and then add page and per_page controls. Notice any unusual helpers? (order_by_link, per_page_select, and page_select). I won’t go into detail about these helpers because its all in the documentation under Searchlogic::Helpers::ControlTypes.

These are only a few ways to use these helpers, this is really just the tip of the iceberg. The sky is the limit: create a select that lets users navigate through the pages, create a list of links (like flickr) that lets you navigate through pages, create a link that lets you order by any number of columns you want, etc. I know ruby has a bad rep when it comes to documentation, but I actually put some decent time into the Searchlogic documentation and I think you will find it helpful. That’s your best resource for finding out everything Searchlogic has to offer.

Adding a search form

“This is all great, but you said I could search my data”. Searching your data is just as easy. Just add this to the top of your index.html.erb

# app/views/users/index.html.erb
<% form_for @search do |f| %>
    <fieldset>
        <legend>Search Users</legend>

        <% f.fields_for @search.conditions do |users| %>
            <%= users.label :first_name_keywords %><br />
            <%= users.text_field :first_name_keywords %><br />
            <br />

            <% users.fields_for users.object.orders do |orders| %>
                <%= orders.label :total_gt, "Has orders with a total greater than" %><br />
                $<%= orders.text_field :total_gt %><br />
                <br />
            <% end %>

            <% users.fields_for users.object.user_group do |user_group| %>
                <%= user_group.label :name_starts_with, "Belongs to user group with name that starts with" %><br />
                <%= user_group.text_field :name_starts_with %><br />
                <br />
            <% end %>
        <% end %>
    </fieldset>
    <%= f.submit "Search" %>
<% end %>

As I mentioned above Searchlogic creates default conditions on your columns based on the type. Letting you use a form builder to call those conditions. When it receives these conditions on the back-end it will do its “magic” and creates the proper SQL. Don’t worry about SQL injections, Searchlogic has you covered on that (see the documentation for more info).

What’s great about this method of searching?

1. Your search logic is in one place: the view.
2. You can add conditions by doing f.text_field “[column name]_[condition]“. So when your picky client calls up and says “Hey, we need a search field for finding emails that end with…” No prob: f.text_field :email_ends_with. Done!
3. You can traverse your relationships with fields_for and set conditions on related objects. Saweet!

Ajaxified

So you’re saying “this is great, but super old school, where’s the AJAX?!?!?” So you’re all about AJAX? No problem. In the example I have 3 examples: a non AJAX example, an AJAX example using the built in rails helpers, and a jQuery AJAX example. Check them out. You can view the source of the examples on github (each example if named spaced into its own controller). On a side note, I highly recommend jQuery. I recently started using it and love it. I actually get excited to write javascript because I always find something new when digging through the documentation or plugins and unobtrusive javascript never felt so good.

Some helpful links

This tutorial is really just the tip of the ice berg with Searchlogic. Checkout these links to see all it has to offer:

• Live example based on this tutorial
• Source code for the live tutorial
• Searchlogic, the library that makes all of this magic happen
• Searchlogic documentation

I am always interested in hearing feedback. Let me know what you think, what you like, what you don’t, that you hate the name, etc. I love criticism. Searchlogic is under active development and I am always trying to improve it. I hope this tutorial was helpful to you and ultimately makes your life easier.

Enough said, let me show you some of my favorite features. If you are still interested you can just jump over to the searchlogic github page for a more detailed explanation.

For the following examples lets assume these relationships: User => Orders => LineItems

Simple Searching Example

User.all(
  :conditions => {
    :first_name_contains => "Ben",          # first_name like '%Ben%'
    :email_ends_with => "binarylogic.com"   # email like '%binarylogic.com'
  },
  :per_page => 20,                      # limit 20
  :page => 3,                           # offset 40, which starts us on page 3
  :order_as => "ASC",
  :order_by => {:user_group => :name}   # order user_groups.name ASC
)

same as above, but object based

search = User.new_search
search.conditions.first_name_contains = "Ben"
search.conditions.email_ends_with = "binarylogic.com"
search.per_page = 20
search.page = 3
search.order_as = "ASC"
search.order_by = {:user_group => :name}
search.all

In both examples, instead of using the “all” method you could use any search method: first, find(:all), find(:first), count, sum, average, etc, just like ActiveRecord.

The beauty of searchlogic, integration with rails

Using Searchlogic in rails is the best part, because rails has all kinds of nifty methods to make dealing with ActiveRecord objects quick and easy, especially with forms. So let’s take advantage of them! That’s the idea behind this plugin. Searchlogic is searching, ordering, and pagination all rolled into one simple plugin. Take all of that pagination and searching cruft out of your models and let Searchlogic handle it. Check it out:

# app/controllers/users_controller.rb
def index
    @search = User.new_search(params[:search])
    @users, @users_count = @search.all, @search.count
end

Now your view.

# app/views/users/index.html.haml
- form_for @search do |f|
 - f.fields_for @search.conditions do |users|
  = users.text_field :first_name_contains
  = users.calendar_date_select :created_after # nice rails plugin for replacing date_select
  - users.fields_for users.object.orders do |orders|
    = orders.select :total_gt, (1..100)
  = f.submit "Search"

- if @users_count > 0
  %table
    %tr
      %th= order_by_link :account => :name
      %th= order_by_link :first_name
      %th= order_by_link :last_name
      %th= order_by_link :email
    - @users.each do |user|
      %tr
        %td= user.first_name
        %td= user.last_name
        %td= user.email

  Per page:
  = per_page_select
  Page:
  = page_select
- else
  No users returned

Things to notice

1. Passing a search object right into form_for and fields_for
2. The built in conditions for each column and how you can traverse the relationships and set conditions on them
3. The order_by_link helper
4. The page_select and per_page_select helpers
5. All of your search logic is in 1 spot: your view. Nice and DRY.

See my tutorial on this example

Nifty methods for searching trees

Just like the conditions you automatically get with each column. You automatically get conditions for searching tree data structures, if you’re model is a tree data structure.

# User is a tree data structure.
conditions = User.new_conditions
conditions.child_of = User.roots.first                  # Finds all children
conditions.sibling_of = User.first                      # Finds siblings
conditions.descendant_of = User.roots.first             # Finds all children, grandchildren, etc
conditions.inclusive_descendant_of = User.roots.first   # Includes root + children, grandchildren, etc.

Any of these conditions can accept a User object or a User id.

Get Started

Check out the GitHub repository to get started:

http://github.com/binarylogic/searchlogic/

Or a tutorial I posted on using Searchlogic in rails:

http://www.binarylogic.com/2008/9/7/tutorial-pagination-ordering-and-searching-with-searchlogic