Erlang Facebook API

I am exploring way to get into user's graph using Facebook API, my language of interest is Erlang. With quick search, unlike Yahoo API, I found couple of erlang module available to do the job. I will go over it and if I find anything cryptic, I will update it on next post.
My requirement

1. User wanted to bring their user graph but not need to provide their userid/password (oauth)
2. Once loggedin, they choose people who they are to be invited 
3. Choose selected one.

This is kind of interesting flow, in fact, others following.  

Constrains,

1. What is the API user agreement?
2. No of calls limit
3. Constrain to post message to Facebook using API
4. Retrieve Events 

I will research these topic and post it here if anything interesting.

[Update] Facebook API doesn't allow contact email addresses of friends but found that there are greater API level integration with different graph API. But clue is "it is easy with oauth" but facebook notoriously stoping me to verify my account using phone no :)  So I am hibernating this effort.

Yahoo oauth and Erlang

I successfully implemented or more aptly integrated yahoo oauth.
Points to note,

• You must sign in and get 
• Consumer key embeds what kind of service you are enlisted 
• Secret 
• App ID (you don't need it for YQL)
• For accessing private data -- it is 3 legged process -- 
• Get request token
• Use that to bring yahoo sign in page to ask user to give authorization to access services 
• Once user accepts, callback URL will be called and with query param 
• Once callback is complete, access actual DATA API using YQL
• YQL determines what service (like contact or connection etc) you are accessing but consumer key tells whether you have that service enabled. 
• If you later add or modify service access, you will get new consumer key
• diagnostics=true is handy to see what is wrong with your request
• Most of the oauth parameters are sent as query string
• Mostly more than 3 requests fired to yahoo to get a data
• window.opener will not work to communicate back to parent window because it is shifting to 2 different domains
• I used ugly timer to track popup window to closing status. I felt window.close event would have been handy.

I used nice little library for erlang (https://github.com/tim/erlang-oauth) 

Erlang DNS TXT, MX record look up

I was exploring how to query DNS and do some probing on various types of information like MX and TXT records.
Here is the simple API
 Step:1         inet_res:nslookup("domain.com", 1, txt).                                                                           
It is always advisable to use another function, to convert it from internal record type to list,
 Step:2        inet_dns:msg(Return_from_Step1)                                                                                                    

 Step:3        inet_dns:rr( Return_from_Step2)                                                                                                           



Step:2 always return list or tuple of error message. So return value from the Step:2 should be processed.




Here is the complete code which returns "TXT" rr record, you can substitute mx for "MX" records,

lookup(Domain, VerifyingText) ->

 case inet_res:nslookup(Domain, 1, txt) of 

  {ok, Record} ->

   DnsRr = case inet_dns:msg(Record, anlist) of 

           DnsRrList when is_list(DnsRrList) -> hd(DnsRrList);

     _ -> []

           end,

   io:format("IS of type ~p ~n", [is_list(DnsRr)]),

   inet_dns:rr(DnsRr, data);

  _-> false

 end.

Pass the domain name to the lookup, you will get the list of records.


 More here http://www.erlang.org/doc/man/inet_res.html

Riak, thoughts on Erlang based client

I compiled Riak and started it, couple of times I tried with documented hello world like example but this time I wanted to separate Riak DB server from Erlang client. Riak is notoriously bad for omitting how to guides -- this also helps early adapter to delve deeply to understand and hence good for devs at the end. I really need to thank Joe Armstrong for pointing to Riak as it is not visible few months back.

1. Riak's default cookie name is "riak" and it can be verified by "ps -ef | grep riak" or rel/etc/vm.args file has cookie properties, here it can be changed to match Erlang client

2. I copied few files from cp <>/riak/apps/riak or luke/ebin to my working directory to check dependent module to work with and found

1. luke.beam
2. luke_flow.beam
3. luke_flow_sup.beam
4. riak.beam
5. riak_client.beam
6. riak_core_util.beam
7. riak_kv_map_phase.beam
8. riak_kv_mapred_query.beam
9. riak_kv_util.beam
10. riak_object.beam
11. vclock.beam

are necessary to talk to Riak using Erlang application

3. Once grap hold of those files and node at which Riak server is running with cookie which Riak uses -- Erlang can talk to Riak period.

4. Now the steps detailed in the README (Riak's) is working fine for me. Assuming you are the directory where the above 5 files are available (or include that dir in erl path (-pa))

I just copied sample code from README file below and io output is omitted for brevity.

prompt> erl -name riaktest@127.0.0.1 -setcookie riak

(riaktest@127.0.0.1)1> RiakNode = 'riak@127.0.0.1'.

(riaktest@127.0.0.1)1> net_adm:ping(RiakNode).

(riaktest@127.0.0.1)1> {ok, C} = riak:client_connect(RiakNode).

(riaktest@127.0.0.1)1> O0 = riak_object:new(<<"groceries">>, <<"mine">>, ["bread"]).

(riaktest@127.0.0.1)1> C:put(O0, 1).

(riaktest@127.0.0.1)1> {ok, O1} = C:get(<<"groceries">>, <<"mine">>, 1).

(riaktest@127.0.0.1)1> V = riak_object:get_value(O1).

(riaktest@127.0.0.1)1> O2 = riak_object:update_value(O1, ["milk" | V]).

(riaktest@127.0.0.1)1> C:put(O2, 1).

I am not sure only these files are enough, but probably not! there are modules for certain kind of map reduce available in Riak, so if you plan to use those builtins you may need to copy them as well.

CouchDb RPC vs HTTP client

I have been lately doing Erlang RPC client for couchdb and found following interesting facts.

Http is almost 3 times slower than RPC and 6-7 times slower than co-located client (client and couchdb running on same beam process).

Details

-------

RPC

runtime=20 wall_clock=280 microseconds

runtime=10 wall_clock=313 microseconds

runtime=20 wall_clock=368 microseconds

runtime=20 wall_clock=402 microseconds

runtime=20 wall_clock=445 microseconds

Http

runtime=80 wall_clock=781 microseconds

runtime=80 wall_clock=819 microseconds

runtime=80 wall_clock=804 microseconds

runtime=80 wall_clock=784 microseconds

runtime=80 wall_clock=798 microseconds

Erlang Websocket on Mochiweb

After long sleepless night and past two days of continuous work, able to run Websocket on Mochiweb. This is going to be checked into Erlwebsockserver. I really welcome folks to try out on Google Chrome which is the only Browser, apart from Firefox --- which has Websocket in its trunk waiting for review borads blessing.

Load testing using tsung

I am trying to use tsung as load testing tool. It is written in Erlang and got advantage of spawning concurrent process easily and so it is best fit for this kind of simulation.

1. I wanted to test CouchDB Multimaster setup and to know how replication can be used reliably in real world scenario.

2. I also want to test HA proxy's TCP based load balancing using CouchDB's native client

running erlang emulator shell as root on mac

I wanted to run couchdb under emulator but the problem with erl emulator is, it is not inheriting or obeying sudo command. I didn't plan well before installing couchdb so that it could run as logged in user. So I tried something like below to make it work on my macbook OS X (10.5.8)
1. sudo bash
2. set ERL_LIBS
3. export it
4. erl
Actual steps are
1. sudo bash (OR) sudo su
2. ERL_LIBS=/usr/local/lib/couchdb/erlang/lib --- path where couchdb is installed
3. export ERL_LIBS
4. echo $ERL_LIBS --- to verify the var is set properly
5. erl It worked finally...

Native Erlang Interface to CouchDB

Http Client
In my application I am using couchbeam as erlang client utility. It is under heavily developed and improved by benoitc. It is one of the excellent http based couchdb erlang client.

Native Client
There is an interface which sits along with couchdb binaries called hovercraft. It looks like not maintained by its developer and it was not working as it is not retrofitted to new couchdb changes. I fixed those things by forking it (http://github.com/sendtopms/hovercraft).

Performance difference
It is native erlang client. So compare to any other http based erlang couchdb client it has advantage of talking to couchdb using erlang terms. So erlang->json ->erlang serialization is avoided. In my ad-hoc testing, I just had a document in couchdb and created view and accessed it through above client.
Couchbeam took 6472microsec - 7331 microsec
Hovercraft took 994 microsec - 1057 microsec.
Will this # matter? It is not humanly noticeable to find the difference but when no of request increases throughput will increase with native view but need to deal with Erlang.

Http client has advantages over native as
1. Easy to loadbalance using HAProxy or nginx or Apache
2. it is dead easy to set up Master to Slave configuration as Load is balanced in the DB request layer so Create/Update/Delete (CUD) can easily be routed to master, in case of native client, LB can be done at front end or presentation layer, but routing to master for CUD is quite complicated if it has session association and state management, but if it is designed and architected from the start it is quite simple to model. After all, native client is 6-7 times more efficient to http based client.

Is Java/C# dead?

Is Java Dead?

Almost a/an year or more back I read about "java is dead, long live java" I felt angry. I felt the author is really lunatic. But ... now I have a reason to say it should die. If you work with Java and struggled to scale for few 100 users... not even for Internet scale you will wish java should die. It looks like "Yes" for web application development. But I differ from original author. Java should die not because its bloat or not SOA ready (SOA is also declared dead by Anne Thomas Manes) but because it inherent issue with Thread and shared model. It can't even be resurrected by models like Ejb to help scale application. It is like mercury and it is hazardous to environment as you need more servers to do the same scale. May be thats the reason IBM left buying SUN.

Who is playing the Game?

Java in late 90s and till 2003 is really good as hardware vendors like Intel focused more on clock speed and tailing directly with moors law. But suddenly they realized that that route won't work and changed the rule. So rule of the game now shifted to multi core. This is the reason software developed for single core with high clock is struggling to cope up with low clock speed with multi cores.

What others Think about Java?

I also found some where in the Internet, I guess it is Steve Yegge, quoted "Java proved even pig can fly". Yes I still felt IBM and other anti MS guys made and give java a Star status. It grown to 800 pound gorilla and become useless in modern Internet scale application. Earlier I was looking to see any of the internet scale companies use Java as their underlying platform. hm... they know what they want.... No where I found a Java in use. I know how complex an Ejb is. But how much it can scale? It is worthless to provide hardware, though it is cheap, to produce sub optimal results -- even cheap will become very costly in the long run. But it is good at certain aspects. I pity on guys like Gavin King who contributed Webbeans and Seam framework which will kneel down and burry itself when 100 concurrent user access it. JCP is useless and they come up with standards when even after somsthing is obsolete, for ex, JSF came and AJAX destroyed it, JSF 2.0 finalized and Websocket and comet or more specific JQuery, Dojo, YUI/Ext Js already doing heavy lifting in the client side. So why we need to do page construction in the server side?

So what is the alternative? Yes there is emphatic answer. It is Erlang. We are in the modern world. We know how Json and REST is useFull and Soap is there to cleanse us from multi language hell. So it Amazon, Facebook, meebo and others enjoyed the power of Erlang. It is like nuclear bomb and it is so powerful concurrency and message passing actor model will help scale.

Erlang - Fileupload

I have been playing with ewgi --- wsgi equivalent of Python. It is spec which is above and top of particular application server likes (YAWS or Mochiweb or inets or misultin) so the application developed for ewgi can run on any server without any modification, ideally.
I have been playing with mochiweb and couchdb and using beepbeep as my application framework. Beepbeep has integration with ewgi and so for good for simple usecase. But I was not able to do file uploading. I added missing fileupload piece to beepbeep but theoretically it should work for any application running on EWGI and Mochiweb. So I forked beepbeep and added my file upload module, you can get it from here http://github.com/sendtopms/beepbeep/tree/master

Erlang and SOAP Webservice

I just completed invoking SOAP Webservice from Erlang.

Here is what I did,

1. inets:start(), This is very important without which http:request will not be succeeded.
2. Took a shortcut: (hm.. not actually) I treated SOAP Call as Http POST call.
3. I used Erlang http module
1. http:request( post, URL, [{"Host" , Host}, {"SOAPAction", ""}],

"text/xml", getData ()},

[ {timeout, 5000} ], [{body_format, binary}])

2. Used case expression to handle different condition
3. getData() returns binary form of soap message. Here, correct payload of SOAP Request will be needed to make this request template. It returns binary form of the SOAP payload (String form can also be used)
4. http:request would return either
1. {ok, Result} -> Handle successful http request
2. {error, Reason} -> Handle Errors
3. {ok, saved_to_file}-> if you use “stream” option, this pattern will be invoked and it is optional.
5. Result is response payload (http payload) and can be processed

Note:

Erlang has good support for Proxy and Proxy authentication unlike python. Python is mess in this area.

Erlang Tools and Modules

I was trying to use Eclipse Erlide but unfortunately I am not able turn it off compiling every time I save the file. It is quite slow and will make you irritated.

So, Since I am again coming from Java factory, thought of finding answers to the following items,

1. Need a good IDE which will be able to give me code completion, need to jump into modules without performance issue
2. How would normally development cycle would be. How would I make a build scripts, it looks "make" is widely used tool.
3. How would be the different kind of environment like dev, testing, production etc organized and automated?
4. I am also started looking at best practices/modules and ways to do Unit testing, XML parsing, Logging, Instrumentation, Socket (TCP/IP) access etc
5. CouchDB Or Mnesia based MySql or other database access.

Erlang - Mac OSX

I have been playing with Erlang on Windows7. I thought of trying with Mac OS X. I found great simple blog which worked like a charm. I installed Erlang from source and all others and it worked.

It also gave me an opportunity to play with command line utilities and tools like Make, SVN etc. This is the piece which I was not able to answer to my friend and now I am comfortable.

Erlang - Some more musing

I am able to see lot of traction and attraction towards Erlang esp. individual contributors active with Erlang. It is good sign. Language will not be success without either community support or big marketing muscle. It is clearly the web giants and startups which anticipate astronomical spike in user growth is focusing on Erlang. It is a scale which is unimaginable and reliability which is dream of architects started using Erlang. To cater such a scale an early decision is very important. Facebook chat application with 10 billion messages and 1200 per seconds and 4+ million active channels is astronomical.

Erlang - Why I need to learn new language?

I am in the process of learning Erlang. Why Erlang? There are couple of reasons. I need to find a language which should internet scale.

Brief about internet scale....

1. Single User application - We developed applications, say MS Office, which caters to single person. We won't bother multiple items here like how much memory it eats up, how much processor it takes, how much disk space it takes etc. Why because it runs on either desktop or laptop; in fact netbooks or palm top (windows mobile based phones, iPhone, blackberry etc) has concern on processor memory and space. Program should be responsive enough for the single user. This is one class application.

2. Minimal User Application - Next, small scale application. where few people access the system and they may not be bothering about whether it is up or restarting to make little bit faster (OS like windows XP needs restart to clean up memory leaks) so here people using are highly tolerant and not the applications as such. So it can run on any normal to medium profile hardware based servers.

3. Enterprise Scale Or Limited User Application: Normally enterprise application like enterprise email, Document Management Systems or leave systems etc will be use by limited set of folks and it is mostly intranet application accessed from local network. Normally it is developed in Java EE, .Net, RoR or Python or combinations of them. Normally Webservice or some other complex distributed programming paradigm will be followed.

4. Super scalable Applications: It is current class application pioneered by Google (Consumer email, youtube, maps), Amazon, Facebook (Chat), current Twitter kind of applications. It was unnoticed and tormented in Telecom environment. All the companies did a really hardwork to come up with their platform which is kind of unique and innovative. Infrastructure is built such a way it scales infinitesimal, internet scaling. It is tough to hit, important events make people hit the website. Recent fiasco Nokia's Ovi store broke down, Tweeter's frequent outage due to spikes. Why the concurrency is not free or not easy. It is like gymnastic or performing martial arts.... needs lots of coordination and control. It is tough and needs practice. Some of the paradigm we used to make it grow like imperative programing (Object oriented programing/ procedural programing) suffered a lot of maintaining state. It is like obese people suffering from excessive fat. It distract many thing including co-ordination. So difficult to parallelize. Ok.. Is there a simple way to over come this. Yes and no. I researched a lot. Closely following Twitter and Facebook gives many idea. First of all needs good hardware and servers. Next needs infrastructure which effectively utilizes the hardware. My desktop and Laptop has dual core (I am 2 years behind extreme computing with Quadcores) and multicore GPU. But how my programming platform effectively utilizes the cores is important point to scale. It is kind of parallel to Car which goes 0-120 miles in 6 secs. What is the use if my speed limit is 30 miles? Or I can say riding a bicycle in a express high way where I can go upto 85 miles per hr. This is exactly it is happening. My hardware is so powerful but my software is so dumb and it is not multi tasking. Or even single tasking with all the cores. It is like 4 people needs to do a job but they don't know how to split it and do the work. So one person starts and others are waiting until the guy finishes. So what is the problem when i have a computer with ample amount of power and I can't normally bother between 0.7 sec or 1.4 secs of completion. Still it is fast.... But what if millions of people access it. It will keel into halt but it suppose not to. So counting every penny is important as it amounts to lot of other things. Energy wastage, environmental issues etc.

So how Twitter or Facebook addressed this issue. They moved some of the heart of the code into Functional Programming Code. For Twitter, it is Scala and for Facebook, it is Erlang. Both are actor based shared nothing function language evolved from Alanso Chruch Lambda Calculus. Academic loves these languages.

So I am in the process of learning Erlang. I found great book Joe Amrstrong's Concurrent Programing with Erlang second edition. So far it is quite interesting. It is lean and mean from the start. It is damed concurrency and fault tolerance and high scalability in its DNA.

There is no loop. What a hell are you talking about? All i need to do is unlearn my OOPS oops!!!.

I create a looping control if i need. I don't need design pattern. I am not telling how it needs to do something. As my friend Santhosh GR said, I just needs to tell what I need. No state to maintain. Oh... how can I program with the one which doesn't have or doesn't recommend one which I used for developing even adding two nos.

Having set the context, I will continue on Erlang later.... It also brings me how Java going to survive? It looks like its current state it is not fit into "Internet Scale". This is same for C# and other imperative languages. Their core is thread. Is thread is bad? It looks, sounds and feels bad. Why because thread are, at least in current state, not able to scale across multiple cores. Java is faster on single core processor with high clock speed compare to multi core highly effective modern processor. It degrades shabbily beyond quad cores and Intel has 6 core already and going to launch 8 cores and AMD story also the same. But Java, Java 6 came 2 years back and 7 still in pre beta. So time and growth of hardware and software are looking north and south.