Kiprosh Team at Spree Conference in Dublin, Ireland

We at Kiprosh had wonderful time attending Spree Conference in Dublin, Ireland and had opportunity to meet Industry leaders, Experts, SME's like Yehuda Katz (member of Ruby on Rails, jQuery core teams), Sean Schofield (Creator of Spree Project and CEO of SpreeWorks), Brian Quinn (CTO at Spree Commerce), Bridget Gleason (VP Sales of Engine Yard), Nick Gauthier (Author of Recipes with Backbone), Paul Cambell (Hypertiny), Gregg Pollack (Envy Labs), Jonathan O'Connor (Code Guru),  Daniel Honig (Architect at DirectNorth), Scott Ford (Cardigan Networks), Lynne Brehmer (Marketing and Sales at Spree Commerce), Antonio Velázquez (Director at Telefonica S.A.), Ruz Chisty (theBusinessPitch), Qamir Hussain (Aveclabs), Ryan Bigg (Co-Author of book Rails 3 in Action with Yehuda Katz), Adil Wali (Kemists) and Andreas Happe (Architect at Starseeders).

Conference was held at World Famous "Guinness Storehouse" in Dublin city.

Here are few pictures from our conference.

Image crawler engine using "anemone + geospider + redirect_follower + memCached"

Recently our team at Kiprosh built an image crawler engine with following requirements (to be strictly developed in 1 week time or less).

1) crawl and spider all images from a given URL (http or https)
2) crawl it as a background process
3) scrape till 3 level deep per link. (configurable depth)
3) save img URL's in DB, caching
4) keep on displaying the crawled images in UI
5) ability to tag these images
6) ability to "multi select" (using shift + mouse clicks) tag and untag images
7) wonderful nice looking UI with ajax, pjax for pagination, tagging and ability to cache
8) search feature based on tags
9) multi size crawl feature

After thorough research and quick PoC, we used gems like geo-spider, anemone, redirect_follower and memcached to build this crawler engine. The overall app turned out to be very stable, scalable, fast and elegant due to usage of these awesome gems. There were other gems in comparison to geo-spider but for our requirement geo-spider served specific purpose to allow retrieve metadata we needed from source URL's. Anemone is another cool gem for depth crawling in URL that other gems and patterns didn't allow us earlier to dive deep into.

Links to these gems and their respective project page
GeoSpider, Anemone, Redirect_Follower, Memcached

On Heroku, we had to use following gems for caching.

#gem "memcached-northscale", "~> 0.19.5.4"
#gem 'memcached-northscale'
#gem 'dalli'

On our dedicated node, memcached worked just fine without customizing or supporting with other versions.

Queue_Classic and Mechanize Gems

Creating a list of useful (indeed very useful) gems for future reference. We used following gems recently in our rails apps (actually products & tools) that we are developing for our clients.

1) queue_classic - Though we have used Redis Resque, RabbitMQ in three to four apps in past but for this specific requirement we wanted to rely on fast, low maintenance message queue providing a simple and intuitive user experience. It is built upon PostgreSQL to avoid the necessity of adding redis or 0MQ on heroku. Yeah queue_classic doesn't increase any database load contrary its pretty efficient due to usage of inherently reliable PostgreSQL methods where PostgresSQL has many wonderful feature such as  Listen/Notify. (even Oracle too supports listen/notify.) Thus, to avoid resque worker running on heroku and due to its sheer simplicity, we opted for queue_classic. BTW, queue_classic is extensively used by Heroku Postgres team to monitor the health of their customer databases processing hundreds of jobs per second.

RailCasts - http://railscasts.com/episodes/344-queue-classic 
Source page and more information is at - https://github.com/ryandotsmith/queue_classic

2) Mechanize - What a wonderful gem we must say :) Well we have done lots of scrapping but mechanize is really handy to automate interaction with websites. We are building a tool (web based) for an enterprise to automate large number of routine and regular tasks for their helpdesk support staff. Mechanize scripts helps us execute these routine tasks that saves a ton of time for the support team.

RailCasts - http://railscasts.com/episodes/191-mechanize
Source page and more information is at https://github.com/tenderlove/mechanize