Spark and hadoop anyone ?

I should have published this a while back - slipped my mind :-)


Spark now supports batch mode on top of Hadoop v2 (Yarn) thanks to the work we did in Yahoo! Labs Bangalore !
The pull requests are here, here, here, here, here and here (to be applied in order). Have been committed to spark master and will be in 0.8 release !
Take it for a spin and try it out on PB of data !!

Sourangshu was involved in this along with me in getting this polished and submitted to spark.

Citibank outbound transfer limit

So this is NOT mentioned (as of now) - but there is a upper limit imposed on Citibank accounts. You cannot make outbound transfers of more than Rs. 15L in one day.

Irrespective of what you try (multiple payees, different transfer types, etc) : the sum total of all outbound transfers in one day cannot exceed Rs. 15L

Unfortunately, this is not documented anywhere : on the other hand, they do explicitly claim no limit to RTGS transfers !


Atleast the relationship manager and others are the branch were super-helpful in trying to find alternatives.

Fedex in Bangalore

Note to self: For delivery of packages within India, DO NOT use FedEx !
They are slower than other courier companies - even slower than Indian Postal service !!

When you are in the courier business, and you cant beat even the Govt. postal service, you really should give up : pathetic service times from FedEx.

And worst part is, on contacting service centre, the conversations roughly go like this :

a) Try to disown the collection centre and claim it does not exist. The attempt is to claim it was not given to FedEx itself.
I personally delivered it to FedEx office on Richmond Road.
When I pointed out the valid tracking number, the call centre guy changed tack.

b) Abuse their employees in the collection centre by calling them incompetent for not sending package in time.
Meh !

c) Backtrack on the SLA given - and claim it was impossible to satisfy.
The lady at the FedEx office was very categorical in stating that it will be received by my dad the very next day : hence the name 'overnight priority delivery' according to her.
And I repeatedly confirmed this.

d) Claim that 'overnight priority delivery' - does not mean you will receive it overnight !!
Apparently overnight means something entirely different.

e) Try to be extremely rude and bluster their way out of it.
Had to cut the call twice due to this - and they refuse to put you through to their supervisor.

f) Feign helplessness and utter stupidity.
"What can I do sir, it is en-route. We cant do anything".



All in all I would say if this is the sort of sub-standard service which can be expected from a "global leader", they might be better off not trying to enter Indian market : particularly considering the exorbitant fees they charge.

Allianz Bajaj life insurance, bad or worst company ?

I have had a couple of life insurance policies with Bajaj Allianz for almost 12 years now, and it has never been a pleasant experience dealing with them ... but recent changes they have introduced take the cake.

The basic attitude seems to be - treat your customers like crooks. They have absolutely no qualm's violating the customers privacy in the name of protecting their interest.

The incident prompting this post goes somewhat like this :



Since it is the time to file tax returns, I was at the local Bajaj Allianz branch in Bangalore to get a statement on amounts paid for last financial year and also pay the next quarterly premiums.

While waiting for the counter to be free (45 mins, sigh), I was observing a customer and her advocate harangued & shouted at by the staff, manhandled by the security 'guard' on instruction of the branch manager and finally thrown out of the branch : after making them wait for an hour - all because they insisted on closing some policy (or some such) ... came across these details given the raised voices with which the staff was publically responding to the customer - so much for discretion while dealing with customer details.
Though I did not interfere, it did leave a very bad taste in my mouth.


And finally, it was my turn - gave policy numbers, intimation letters and cheque to the cashier : and suddenly I see the cashier bring a black object near my face - was quite surprised at this, and quizzed him about what was happening.
His response ? "I am taking your picture" ... what the hell ?!

Apparently violating customer privacy, or having minimum decency of asking permission before taking a picture is not something the company officials think is important.
I refused permission to take my photo and offered to show a valid photo-id in case he wanted to validate it was my account (I offered driving license and passport - both which carry my picture) - and his response was  "I cant give you the statement without allowing me to take your photo" !
On insisting on talking to his supervisor, he directed me to the manager - her arguments were weird, nonsensical and plain fantastical on why they need to take the photo : Govt issued photo-id is not valid apparently, since I might look like my sister - go figure, considering she is 4 years younger, and I am a guy :-)
Bottomline was, apparently their 'head office' has issued instructions some 4 months back asking all branches to do this. In the name of protecting my privacy, they want to violate my privacy !
On asking her details about who authorized this rule, she was clueless - since, AFAIK, the insurance regulatory body has not issued anything of this sort (pls correct me if I am wrong).

If you think they were really looking out for the customer, you will like what came next.
Before leaving the branch, I asked the cashier for other details about my policies (when they will mature, when was the last payment made, when was the next one due, etc) - and he very readily gave them to me.
So this entire exercise of protecting customer interests is just hogwash ... why they need to harvest these details is something I am not sure of ....


What was really contrasting was that I had just gotten statements from my Bank before - and did not even need to furnish a photo-id proof (they could validate who I was based on KYC details they have).



Allianz Bajaj Life Insurance really needs to get its act together ... forget customer friendly - they need to start treating customers with more dignity !
Personally, I wont be dealing with them in future.

Scala weirdness

This one almost cost me a weeks work, and you are going to love it - maybe obvious to Scala practitioners - but definitely unintutive to someone who is just using Scala : like me.


Consider a class

class MyClass(private var field: Int) {
  private def processField(value: Int) {
    // For now, assume it is just setting it after some processing
    this.field = value
  }
}

Do a javap on it, and we get :

> $JAVA_HOME/bin/javap -p  -cp . MyClass
Compiled from "MyClass.scala"
public class MyClass implements scala.ScalaObject {
  private int field;
  private int field();
  private void field_$eq(int);
  private void processField(int);
  public MyClass(int);
}

Which is fine - our method processField exists : with same signature as we would expect if we had written it in java.


Now, let us add an object MyClass - which can directly invoke this processField. For completeness sake, I am including the modified .scala file below :

class MyClass(private var field: Int) {
  private def processField(value: Int) {
    // For now, assume it is just setting it after some processing
    this.field = value
  }
}

object MyClass {
  def someMethod(inst: MyClass, value: Int) {
    inst.processField(value)
  }
}

Running scalac on this causes no errors - as expected : the object can invoke private methods on MyClass instance - as scala defines.
But now, there is a non trivial change to the generated class. Notice it below.

> $JAVA_HOME/bin/javap -p  -cp . MyClass
Compiled from "MyClass.scala"
public class MyClass implements scala.ScalaObject {
  private int field;
  public static final void someMethod(MyClass, int);
  private int field();
  private void field_$eq(int);
  public final void MyClass$$processField(int);
  public MyClass(int);
}

Yep, the processField method is now NOT what we defined :
a) It has been renamed with namespace prefix.
b) It is now PUBLIC ! Yes, a private method has its access permission changed !

(b) is a very severe issue : and probably a bug IMO and (a) is what hit me as a issue



Now you ask, why all the fuss about all this ? After all, the names are private to scala and invocations to them work as expected.

Yes and no - when you are interfacing java with scala, things will fail.
And more importantly, when you are interfacing scala with java - explicitly or implicitly, things can fail.

The last is what happened to me - one word : SERIALIZATION.

I was invoking readObject/writeObject from the object - which, from a java background, is common practise when you have custom serialization (to invoke from a static method).

Here, it fails spectacularly - since the method signatures are royally mangled by scala.


So the dirty workaround ?

- Add all the actual logic within readObjectImpl and writeObjectImpl
- readObject and writeObject delegate to the Impl methods.
- object invocations are on the Impl and NOT on read/write Object

This preserves the method signature while ensuring things work.
I found this out the hard way - I hope there is an easier solution, but I did not find it.


In the hope that someone else might not trip over this issue and waste days, posting it here !

JDK 7 weirdness !

This is completely unintutive behavior in JDK 7.
I am sure the language theorist's will have some reason why it makes perfect sense, but as a programmer, it is confusing and unintutive.

Consider this simple class :
public class Jdk7Bug {

  private final int value;
  public Jdk7Bug(int value){
    this.value = value;
  }
  public static <C extends Jdk7Bug> void bug(C instance){
    System.out.println(instance.value);
  }


  public static <C extends Jdk7Bug> void works(C instance){
    Jdk7Bug val = instance;
    System.out.println(val.value);
  }
}

As you guessed it, the method 'bug' will fail with compile error in jdk7 (not in 6).
Generics is complicated enough without introducing weirdnesses like this, sigh ...

AMD #Fail !

For a company making a lot of hype and noise around GPGPU computing AMD/ATI has done a piss poor job at it.
AMD has been going to press a lot about OpenCL and how it is next best thing to sliced bread : and yet their own drivers are functionally neutered. Some of the primary benefits of OpenCL are lost due to lack of support for concurrent kernel execution ! I am told that this is a purely driver issue since the newer hardware (6800 series which I have) is supposed to support it !!

Yep - with hundreds of processors on the card, and yet we can still execute only a single kernel at a time because engineers in AMD are not pushed hard to add support for this in the driver : this is particularly sad considering that Nvidia has had this support for quite a while now ....
And to make matters worse, it is rumored (from the forums) that the next version of their SDK and driver will still NOT contain this support ...
Considering that users have been raising this as early as 2009, this is definitely not a new demand !
For a company which is betting heavily on GPGPU (AMD fusion APU, etc), this is definitely not an encouraging sign.



Sigh, talk about wounding customers and then sprinkling salt liberally over the wound !


Now considering if it is worth spending money on a Fermi nvidia card .... so soon after buying the ATI card :-(

My favourite pic

I adore this pic :-)