Seven years ago, when I still worked as the Internet Architect for I-Net Bridge, a company distributing Market Data (real-time stock information and news) in the South African market, I went to my boss, Paul Septhon and said that we had to extend the real-time messaging layer (IML) to include ASCII style messages so that it could be easily integrated into I-Net’s web delivery platforms.

IML (I-Net Bridge Messaging Layer) as it was called at that point, was a publish/subscribe real-time messaging layer for distributing I-Net’s real-time data to it’s customers, from various data sources such as the JSE, Bridge, Dow-Jones and the London Stock Exchange.

The problem was that the publisher and subscriber API’s were extremely event-driven, using callbacks and largely implemented using C, or C++. When it came to developing our web applications it became a problem to integrate a call-back driven, and binary-transport focused system into web applications that are typically “request-get-forget” style systems.

Thus, was invented “CABS” aka “Common Application and Backoffice System”. CABS predated service-oriented architecture and distributed systems that we are seeing now, by about 6 years. Using the existing reliable binary-focused publish/subscribe system that was IML, I-Net developed a scalable ASCII-protocol based client/server architecture that makes things like gearman look like amateur attempts.

The system support load-balanced function calls, a complete directory-like tree structure, mount points for various publishers and a plethora of client and publisher interfaces, including TCL, php, Perl and C/C++.

Data could be accessed transparently in the entire “data” tree, with full ACL based permissions required by the underlying IML layer, thus limiting the access of data by clients only to publishers that they subscribe to. Publishers could then implement finer grained access control. We proceeded to implement one of the most feature rich, web-based MDDS syndication and publishing systems in South Africa based upon this architecture.

It was a phenomenal achievement and I reckon, one of the grandest in South African development history, considering the time, the recent .com bubble bursting and everything that ensued post-that. We even implemented user-authentication and statistics gathering using this architecture. We had about 8 Apache based-linux front-end servers, communicating with the “cloud” of distributed data publishers across multiple geographic locations.

The front-end apache’s were mod_perl and HTML::Mason scripts that talked to the publisher’s with a simple ASCII style protocol. The HTML::Mason components used aggressive memcached caching in order to scale our performance.

Nowadays, I hear about “Web 2.0” startups, and dig into the architecture and system used, and have not found anything approaching the implementation we had at I-Net Bridge.

Until, today I came across gearman. Having been a memcache and danga.com fan for many years, I was surprised to see — finally, something that resembles the original I-Net Bridge CABS.

Gearman, is very simple, based on a simple job submission client, “mnemonic function” based job-router (gearmand) and hooks up to a bunch of “workers” that actually do the work.

In terms of architecture it focuses on the basics, redundancy, scalability and leaves all the rest of the complicated stuff such as the actual handling of access-control and marshalling of data as a “undefined contract” between the publisher and subscriber. Gearman simply handles the distribution, and reliable queuing of tasks and responses. It doesn’t even have client authentication! Those, I can work around fairly easily…

It is nowhere near as complicated as CABS was (nor do I think it will ever be) but having waved a sad good-bye to an amazing system at I-Net Bridge, I’m glad to finally find something that allows me to build some systems on a common distributable platform. I’ve been fiddling with PHP beans, UDP-based broadcasting of requests queues and various other solutions for Neology‘s carrier-grade caching, RADIUS and billing systems, and I’m glad to have finally found some replacement “glue” to get everything together again in a consistent fashion.

I intend to use gearman for everything, including pinging my desktop 🙂

I recently (tonight, actually) had a requirement for the usual set() get() and incr() memcache style key-based variable access in PHP (on Linux) without the usual mucking about of installing memcached, and php5-memcached on Debian. I’m busy writing some modules for OpenRADIUS and they need to be able to increment some shared statistics counters, and share other state information between multiple instances of the same module.

I didn’t need cross-machine sharing of the data, and needed a quick (fairly platform independant) way of sharing some variables between processes. Previously, I relied on memcache, but the socket and stream i/o overhead in a high-performance solution is simply too much for the simple sharing of a hashtable I require.

As any good linux programmer will know, shm_* is your friend. SYSV IPC and shared memory is a fairly light, mostly kernel-based implementation mechanism to share memory across Unix processes.

Without further ado, the class:

// This code is public domain
// Original author: Roelf Diedericks (rodent@rodent.za.net)

class sharedMemoryStore {

	private $shmk_key;
	private $shm_id;
	private $var_key=1;

	private $sem_id;

	public function __construct($key="",$size=0,$perm=0666) {

		if ($key=="")
			$key=__FILE__;

		// default 16KB size shared memory
		if ($size==0)
			$size=1024*16;

		$this->shm_key=$key;

		$this->shm_key=ftok($key,'N');
		$this->shm_id=@shm_attach($this->shm_key,$size,$perm);

		if ( empty($this->shm_id) )  {
			throw new Exception("shared memory allocation failed");
		}

		$this->sem_id=@sem_get($this->shm_key,1,0666,true);

		if ( empty($this->sem_id) ) {
			throw new Exception("sem_get failed");
		}
	}

	public function lock() {
		if ( !sem_acquire($this->sem_id) ) {
			throw new Exception("lock failed");
		}
	}

	public function unlock() {
		if (! @sem_release($this->sem_id) ) {
			throw new Exception("unlock failed");
		}
	}

	public function set($key,$value) {

		$this->lock();
		$res=@shm_get_var($this->shm_id,$this->var_key);
		$this->unlock();

		if ($res===FALSE)
			$res=array();

		$res[$key]=$value;

		if (!  shm_put_var($this->shm_id,$this->var_key,$res) ) {
			throw new Exception("shm_put_var failed");
		}

	}

	public function get($key) {
		$res=@shm_get_var($this->shm_id,$this->var_key);

		if ($res===false) {
			echo "warn array empty\n";
			return false;
		}

		return @$res[$key];
	}

	public function incr($key,$increment=1) {
		$this->lock();

		$res=@shm_get_var($this->shm_id,$this->var_key);

		if ($res===FALSE)
			$res=array();

		if ( empty($res[$key]) )
			$res[$key]=0;

		$res[$key]+=$increment;

		if (!  shm_put_var($this->shm_id,$this->var_key,$res) ) {
			$this->unlock();
			throw new Exception("shm_put_var failed");
		}

		$this->unlock();

		return $res[$key];

	}

	public function  __destruct() {
		$this->unlock();
	}

}

Using it is as simple as the following:

include_once("sharedMemoryStore.php");

$s=new sharedMemoryStore("some-identifier");

$s->set("foo","bar");   // set key "foo" to value "bar"
echo "getting foo:";
echo $s->get("foo");        // get the value of foo
echo "\n";

$s->incr("counter",2); //incremement "counter" with 2
$s->incr("counter"); //incremement "counter" with 1

echo "counter is now: " . $s->get("counter") . "\n";

Now, if you fire up multiple processes on the same machine, they can all share the same counters, or hashtable. The shared memory segment is appropriately locked, so that only a single process can access the variables at a time. incr() works atomically, as expected, so that multiple process can increment a counter without treading on each others’ toes.

The size of the shared memory segment is by default a bit small (16kb), because most Linux distributions don’t have decent shared memory defaults. If you need to increase the size, check the appropriate sysctl.conf setting for your distribution, and change the constructor to something like

$s=new sharedMemoryStore("some-identifier",1024*1024; //1 meg shared segment

I went from about 300 requests per second to 1500 requests per second on a low-spec virtual machine by simply dropping in the shared memory storage class.

MyADSL recently wrote an article about “Free proxy service fraud allegations“. Basically some guy from George, apparently, named Zaine Lourens installed a PHP based web proxy, firstly on a server at Hetzner, then at Afrihost, and then finally at Elitehost.

I read thru the entire “thread” where Lourens originally posted information about his “free” international web proxy.

The amount of social engineering that was enacted by this guy is actually pretty amazing. He fed off MyADSL forumites’ hate for Telkom and being capped and only having local access.

He turned that thread into his own personal glorification field day and I can see how things went from bad to worse over the course of a month. Nearing the end of August, when both Hetzner, Afrihost, and Elitehost had finally kicked him off their servers for breach of Acceptable Use Policy  he simply started pasting the IP adresses of some REAL open proxy servers listening on port 3128. Placating forumites with “I’m checking it out” (when they inevitably went down) and inferring that they are somehow “his proxies”.

He had to keep feeding his ego somehow, because everyone was just calling him “Mr Awesome”. And the reaction of people in the forum was even more shocking. Of course MyADSL’s userbase contains all kinds these days, so I guess it’s only natural that something like this happened, and then actually got some airtime on MyADSL in the form of an article.

Unfortunately, now, people are crying because fraud has been perpetrated, donations have been sent to a fraudster, and waaaaah all around. If it’s too good to be true, then chances are that it’s too good to be true. If someone offers me something for free, my geneneral first response is: “What’s wrong with it?”

Enough about Lourens though, it is on these open proxies that I want to focus for a bit. The info  is all publically available information from the posts.

196.41.132.28 (cte-cache.vwol.net)
Hosted by: MWEB
Type:Netapp/Netcache
FAIL: 8080, 8081, and 3128 left open with no ACLS or authentication.
Status: Looks like ACL’s have been applied.

196.41.26.122
Hosted by: Datapro
Type: CentOS
FAIL: Squid/port 3128 left open by admins with no ACLS or authentication
Status: Looks like 3128 is now being filtered.
Currently: mysql and another bunch of stuff quite open.

196.41.26.123, and 196.41.26.124
Hosted by:Datapro
Type:FreePBX boxes
FAIL: squid/port 3128 left open by admins with no ACLS or authentication
Status: Looks like 3128 is now being filtered
Currently:  mysql, webmin and others still wide open.

The FreePBX boxes are weird. What are they doing with an Open proxy installed? Is this a default thing ? Why do you want squid on a PBX ?

In conclusion:

Someone told me the other day that Africa, and South Africa isn’t really prepared for this “true  broadband” and “loads of bandwidth landing on the continent thing”.  If we consider that these proxies were well-abused over the course of a month, and were probably copied and pasted from  some standard “Open Proxy List” off the Internet, or even just discovered using nmap, then I have to say that that statement is probably holding true.

Once hackers, and script kiddies get nice low-latency access to South African data centres, they’re going to have a field day. And I reckon most local companies simply aren’t ready for it. Go get some kind of security certification now and I reckon in a year or two’s time you’ll be earning top dollar.

Joachim Witt. If it’s the only new artist to ever get introduced into your limited repertoire… Ever. Please, just listen to him. I can’t directly link to the man’s music, nor do want to “induce you” to download  it. But buying just him, is a mission impossible.

But worth it.

I have a taste for German Neue Deutsche Harte,  Bauhaus, and   Architecture… And Joachim Witt is simply where it all started, musically, and genre-wise.  I  will not comment about the german girls playin guitar, since there is no need. It “augments”.

On top of the usual video artistry (did Anton Corbijn direct this video?) ,  Joachim Witt is a master of song, reinvention across decades, a depiction of reality, and a whimsical reflection upon the 80’s, life and 42.

Amongst my genre-favortite bands such as Rammstein, Kraftwerk, and Oomph, and Wolfsheim you will find that Joachim Witt has been the “grandfather”. The literal inventor of NDH. You may want to listen to the REALLY 80’s versions of the songs, and reflect where German NDH has come from and where it’s gone.

Joachim Witt tracks to search for:

1. Batallion D’amour
2. Goldener Reiter
3.  Weh-Oh-Weh.

I used to directly link  to  my “https://rodent.za.net/me/” which had some samples and a wiki-like description of my “Musical Education”  pages before, but all it got me was take-down notices. Funny that I can link to the youtube video’s without problems…

I’m sure that 90% of the bands  I used to “educate” people  with  would have actually appreciated the attention.

Oh well. Welcome to new-age media. Where the artist wins… Or NOT.

If, like me, you were brought up in Pretoria, South Africa then you most likely encountered loads of camping trips endowed upon you by your parents in the misguided belief that it would be “fun for everyone”. Aside from the fact that camping is of course, cheap. For me, at the time it felt that all we were doing is visiting boring after boring, dusty after dusty venue with nothing to do but read.

If, like me, you were also forcefully endowed into the abysmal system of slavery that was called ‘conscription’ in the South African National Defence Force, you probably encountered other kinds of “camping” trips that made the experience from childhood seem like a walk in the park.

All of this, instilled in me a complete sense of dread and adversity whenever camping was involved. Over the last years of my adulthood, I’ve simply shunned all forms of camping as “sub-human”.

So, here I am 20 years later, having actually enjoyed a camping trip. Vincenzia’s requirement for her 32nd birthday was a simple one, yet for me (initially) nearly unachievable due to my preconceptions.

She wanted an “adventure”. With Ruben now at the age of seven, and really having developed into a true rascal, my years of shying away from camping was bound for an overhaul. Besides, I’d actually bought a tent about a year ago so with the idea that Ruben could have some fun with it.

So, Ruben and I google’d mom’s secret adventure, and landed onto the website of Hartebeespoort Oord, camping and otherwise average-looking resort.

Aside from the fact that there was a (apparently unsuccesful) Christian Rock concert scheduled for the entire weekend (have you _ever_ heard of something as oxymoronical as Christian Death Metal?) the weekend was a blast. Ruben used his scooter in the pretty impressive skate park, we played mini-golf, swam for hours in the heated pool, and just generally relaxed.

I stuck to a few basic rules though:

1. Go prepared. In fact, go overprepared.
2. Go somewhere where there is LOTS of green grass.
3. Go somewhere where there is LOADS of things to do for a 7 year old kid.
4. Limit the damage by going somewhere close, and only going for one night (grin).

In all, it’s turned out  to be a complete blast (again, aside from the Christian Death metal).

I guess sometimes  you have to “get out of it”  a little bit in order to appreciate things back at home, and to see what the rest of the world is doing.  It also takes you out of your comfort zone, away from the drudgery, and just this  simple act, of 36 hours  has taken nearly a million miles off my stress-ridden shoulders. Vincenzia was entirely delighted with the birthday “present” and has already started planning another million trips I’m sure…

I might try this again, in a few months time. Time to start un-turtling… (Thanks Joe). If only there was a site somwhere on the interweb’s where people could rate their experience truthfully. Hmmm. Maybe I shoudld consider registering the24trip.co.za …  😉