Using data objects between PHP and Perl via Memcached
Came across a new problem. The desire to share data objects between Perl and PHP via Memcached. This is a bit of an issue since the languages serialize data in different ways and I either have to force them both to conform to the same or modify one to deal with the other. So I decided to use JSON since there is easy support in both languages (and fairly fast parsers).
Granted the easy solution to this is anywhere in code that you want to move data back and forth just use json calls and convert your data to a string and then store it. But want if we want to use OOP and not have to worry about eveywhere in code that already calls existing code? So you'd have to modify the code that creates the inital underlying object that you're wrapping.
For Perl I'm using Cache::Memcached::Fast due to it's performance and ability to easily override functionality.
For PHP I'm using the Memcached library because I think it's much better than the Memcache library that seems to commonly get installed.
Now in theory this should be fairly easily, set both sides to use JSON as the serializer and let'r rip. Turns out not so (man I wish things were easy). In Memcached when you set an object you send in four things. Key name, flags, ttl, and data size. The PHP and Perl libraries handle flags totally differently. Perl took the approach of 1/0 where 0 is string and 1 is serialized (if you tell your object to use the wrong one it's your problem). PHP on the other hand has a hard coded list of serializers and sets the flag to whatever is appropriate (6 in this case for JSON).
So for strings, yes it was quite compatible. But where I needed it there was a total failure (strace showed both were doing the proper JSON encoding on the data).
After some deliberation I decided to modify the PHP side (Perl is our primary language, so gimp the less used one). I decided to add a new flag for the Memcached object that worked in conjunction with the JSON support. Basically if doing JSON and you set the compatability flag you don't set the flag to 6 but to 1 instead to match Perl. This way both clients think that it's serialized and decode/encode data correctly.
I looked at an Erlang library for Memcached (another language we use) and found they were using a random Int for the flags in memcached. Still not sure what the point of that was, but perhaps I'll have a Erlang hack too in the future.
So the zip includes the two patch files for the Memcached source and two quick test files. One is Perl and the other is PHP. They assume that you have a local Memcached server running on the normal port. You should see something like this:
[root@host (13:40:16) /local/tmp]# perl -f local_memcached_test.pl
Setting string
$VAR1 = 1;
Getting same string
$VAR1 = 'This is a string';
Setting hash object
$VAR1 = 1;
Getting hash object
$VAR1 = {
'c' => 'yo mama',
'a' => 2342342
};
Setting array object
$VAR1 = 1;
Getting array object
$VAR1 = [
'b',
2342,
'e',
'poopers'
];
Getting php string
$VAR1 = 'this is a string';
Getting php object
$VAR1 = {
'c' => 12,
'a' => 4
};
[root@host (13:40:21) /local/tmp]# php -f local_memcached_test.php
Have JSON: int(1)
Setting string
bool(true)
Getting same string
string(16) "this is a string"
Setting assoc array object
bool(true)
Getting assoc array object
object(stdClass)#2 (2) {
["a"]=>
int(4)
["c"]=>
int(12)
}
Setting array object
bool(true)
Getting array object
array(4) {
[0]=>
string(1) "d"
[1]=>
int(12323)
[2]=>
string(1) "e"
[3]=>
string(8) "advwewge"
}
Getting Perl string
string(16) "This is a string"
Getting Perl object
object(stdClass)#2 (2) {
["c"]=>
string(7) "yo mama"
["a"]=>
int(2342342)
}
So now I never have to do all of the extra encoding outside of the objects, they just work together. Lazy++