Riak a super scale datastore

I started looking into Riak lately to see how it can fit into my application which is currently using CouchDb. I really like CouchDb for

1. Map Reduce view -- just throw some mr function you will find out the document which it preserved

2. Web UI (the futon) it alleviate lot of Admin pages along with lately added security is good enough to manage it

3. Hot data backup, it is an excellent feature and makes like easier

Cons:

1. Mindset of treating CouchDb as all in all self contained product and hence lacks native interface, (there are APIs but it is based on http -- even http world is moving to websocket due to its inherent limitation for certain usecases, having Database served by http to another middleware is costly and environmentally hazardous like gas guzzling trucks

2. lack of native search (full text) but there is a lucene based one but already couchdb is dependent on JS, Erlang, ICU and having Java into the fold is kind of nightmare to the mix but it works and also gives head ache when replicating and spinning multiple nodes

3. It is not truly distributed but it is kind of storage engine which can be equated to better dets storage engine or innostore but there is nothing similar in couch architecture to be comparable with Cassandra or Riak.

CouchDb RPC vs HTTP client

I have been lately doing Erlang RPC client for couchdb and found following interesting facts.

Http is almost 3 times slower than RPC and 6-7 times slower than co-located client (client and couchdb running on same beam process).

Details

-------

RPC

runtime=20 wall_clock=280 microseconds

runtime=10 wall_clock=313 microseconds

runtime=20 wall_clock=368 microseconds

runtime=20 wall_clock=402 microseconds

runtime=20 wall_clock=445 microseconds

Http

runtime=80 wall_clock=781 microseconds

runtime=80 wall_clock=819 microseconds

runtime=80 wall_clock=804 microseconds

runtime=80 wall_clock=784 microseconds

runtime=80 wall_clock=798 microseconds

Load testing using tsung

I am trying to use tsung as load testing tool. It is written in Erlang and got advantage of spawning concurrent process easily and so it is best fit for this kind of simulation.

1. I wanted to test CouchDB Multimaster setup and to know how replication can be used reliably in real world scenario.

2. I also want to test HA proxy's TCP based load balancing using CouchDB's native client

running erlang emulator shell as root on mac

I wanted to run couchdb under emulator but the problem with erl emulator is, it is not inheriting or obeying sudo command. I didn't plan well before installing couchdb so that it could run as logged in user. So I tried something like below to make it work on my macbook OS X (10.5.8)
1. sudo bash
2. set ERL_LIBS
3. export it
4. erl
Actual steps are
1. sudo bash (OR) sudo su
2. ERL_LIBS=/usr/local/lib/couchdb/erlang/lib --- path where couchdb is installed
3. export ERL_LIBS
4. echo $ERL_LIBS --- to verify the var is set properly
5. erl It worked finally...

Native Erlang Interface to CouchDB

Http Client
In my application I am using couchbeam as erlang client utility. It is under heavily developed and improved by benoitc. It is one of the excellent http based couchdb erlang client.

Native Client
There is an interface which sits along with couchdb binaries called hovercraft. It looks like not maintained by its developer and it was not working as it is not retrofitted to new couchdb changes. I fixed those things by forking it (http://github.com/sendtopms/hovercraft).

Performance difference
It is native erlang client. So compare to any other http based erlang couchdb client it has advantage of talking to couchdb using erlang terms. So erlang->json ->erlang serialization is avoided. In my ad-hoc testing, I just had a document in couchdb and created view and accessed it through above client.
Couchbeam took 6472microsec - 7331 microsec
Hovercraft took 994 microsec - 1057 microsec.
Will this # matter? It is not humanly noticeable to find the difference but when no of request increases throughput will increase with native view but need to deal with Erlang.

Http client has advantages over native as
1. Easy to loadbalance using HAProxy or nginx or Apache
2. it is dead easy to set up Master to Slave configuration as Load is balanced in the DB request layer so Create/Update/Delete (CUD) can easily be routed to master, in case of native client, LB can be done at front end or presentation layer, but routing to master for CUD is quite complicated if it has session association and state management, but if it is designed and architected from the start it is quite simple to model. After all, native client is 6-7 times more efficient to http based client.