FreeRADIUS is a great RADIUS server, but custom integration with it has pretty much always left developers with the choice of rlm_python, rlm_ruby, or rlm_perl.

Understandably, rlm_python is not an option for me, so I’ve been doing some work with rlm_perl, and pretty much the first thing I did there, was to use Gearman, and serialisation to get a FreeRADIUS request out of perl as quick as possible, into a gearman queue, and then and into my chosen worker platform of choice — PHP.

Besides, does one REALLY want to drag an entire interpreter, business logic and it’s potential pitfalls into the FreeRADIUS core, and potentially affect it’s stability ?

FreeRADIUS also makes heavy use of threads, so to integrate a random language is quite a bit of a pain, and most of the language modules are non-thread safe so you cannot really benefit from FreeRADIUS thread scaling.

I don’t mind perl, and I’ve worked in it for long enough, but once one looks at the bridge that the above builds to get into a messaging layer, it kind of makes one think… Can it be better ?

Enter rlm_zmq

I’ve built this FreeRADIUS module over the last two days, firstly, because I haven’t really played with zeromq before, and secondly because I wanted to see if I can make the bridge a bit shorter, and faster.

Well, the results are pretty cool. 4500 requests per second for authorisation only in FreeRADIUS, and 3500 requests per second if I use rlm_zmq in every possible processing section in FreeRADIUS (such as accounting, post-auth etc).

ZMQ still requires a “broker” or “queue” manager of some kind if you’re doing the sort of fan-out to fan-out of multiple worker thing that’s required to get some nice scale.

The basic zmq socket architecture is request reply, along these lines:

freeradius_rlm_zmq(zmq socket pool) ----> queue.php <------ worker(s).php

queue.php is a simple message router, that routes between FreeRADIUS threads and worker processes.

In essence, very similar to a standard gearman architecture, but much more ‘native’. ZeroMQ sockets pretty much behave like sockets, except a bit more clever. Gearman has a lot more functionality than zeromq, and I might just attempt rlm_gearman next.

Either way, gone are the days of having to build language specific modules for FreeRADIUS, I think. If you can deserialize json, and can use any of the many language bindings for ZeroMQ, you can now do RADIUS.

rlm_zmq basically adds Mongrel2-like functionality to FreeRADIUS, giving you the option of 30+ languages and a number of N-to-N messaging patterns and load distribution capabilities.

rlm_zmq is pretty much alpha, but it’s fairly stable, and hasn’t yet eaten all my memory alive after hours of hammering with radperf which is always a good thing. In fact, I think it’s about as stable as rlm_perl 🙂

This was a fun project with pretty cool results.

If you’ve ever tried to use Devel::Leak to debug some perl memory leaks on debian, and found that sv_dump doesnt actually output anything useful for you then try the following:

apt-get install debugperl

then, download Devel::Perl from cpan, untar and build with the following magic:

debugperl Makefile.PL DEFINE=-DDEBUGGING

As  Brendan O’dea (or bod of l2tpns fame) from down under puts it quite succinctly:

The perl-*-debug package provides a debug-enabled binary as debugperl. It does not however include either the config.h or Config.pm which were which match that binary. This is not generally a problem if you just want to debug from the command line (“debugperl -D…”).

Devel::Leak however uses the former (config.h) in the .xs to determine
whether or not to use sv_dump, and the latter (Config.pm) to warn about
-DDEBUGGING in the Makefile.PL.

 

Hopefully this helps someone else googling.

So, I’ve been working as ever, hard on getting some stuff to scale.

For example Lusca/Squid.

I’ve already done the non-threaded async event driven loop, for neoGUARDIAN, our transparent SMTP cunning gadget, and it’s been doing great snooping on SMTP traffic to find out if you’re a spammer or not.

I used Danga::Socket and some clever linux ioct’ls to get full transparency incorporated, meaning that you wouldn’t even know if I’m snooping on your TCP port or not. Interestingly enough, the interpreted perl is not whatsoever the bottleneck. I’ve been quite impressed with the scalability of the thing, especially when offloading the hard work to a Gearman style worker asynchronously.

Now, however something else has grabbed my attention.

The linux splice() and tee() syscalls.

Imagine a file descriptor, hooked up to a client socket, and another to a server socket. Typical relaying proxy style.

Now, instead of having to read() from one copying stuff to a buffer and then write()ing it to another buffer, you can simply tell the linux kernel to connect the two socket handles together, and the kernel will take care of connecting the source and destination without any user space copying.

Now add a tee() similar to the Unix “tee” command, and you can write the data being sent and received by  the two sockets to disk. In fact, they don’t even need to be sockets.

Can anyone say caching proxy ? Lusca has really made some strides in breaking out this kind of code so that it can be modularized, but it’s still going to take some serious hacking because the old squid code base really does depend so much on memcpy()ing things around.

The splice() functionality also supports splicing()ing from virtual memory and other kind of neat things.

It’s in my mind the ultimate way to get to the ZeroCopy style of I/O.

Certainly, portability is an issue, but then again, most modern systems start emulating Linux syscalls simply due to their elegance. It wasn’t always so, in fact Linux emulates a lot of BSD/SySV style stuff, but that comes with the Unix domain…

The question is… Is portability in code worth the performance penalty?

I’m almost keen to start a Lusca Linux-only port. Quite frankly I haven’t given a toss about portability ever, and I’m sure it would probably end up being a better product. Take nginx for example. Yes it will run on win32, but it sucks.

If your Operating System  kernel doesn’t support some standard features required to get scalability then I’m sorry, but you’re fucked. Why should developers have to #ifdef crap just to make sure it compiles on some ancient thing?

The Java fanbois will probably jump at this opportunity to indicate the portability and awesomeness of their environment, but all I can say is that not living close to the kernel will always mean that you’re screwed due to the happy “compile once run anywhere” mantra. Good luck in “import System.Socket.Splice” or whatever it might be called in Java.

For me, it’s simple, either port it to your kernel or whine at your vendor (futile, isn’t it?)

In my mind it’s time to pick a platform and let the kernels match the portability. The effective implementations will survive. The crap ones will die.

Natural select()ion for the win.

I recently (tonight, actually) had a requirement for the usual set() get() and incr() memcache style key-based variable access in PHP (on Linux) without the usual mucking about of installing memcached, and php5-memcached on Debian. I’m busy writing some modules for OpenRADIUS and they need to be able to increment some shared statistics counters, and share other state information between multiple instances of the same module.

I didn’t need cross-machine sharing of the data, and needed a quick (fairly platform independant) way of sharing some variables between processes. Previously, I relied on memcache, but the socket and stream i/o overhead in a high-performance solution is simply too much for the simple sharing of a hashtable I require.

As any good linux programmer will know, shm_* is your friend. SYSV IPC and shared memory is a fairly light, mostly kernel-based implementation mechanism to share memory across Unix processes.

Without further ado, the class:

// This code is public domain
// Original author: Roelf Diedericks (rodent@rodent.za.net)

class sharedMemoryStore {

	private $shmk_key;
	private $shm_id;
	private $var_key=1;

	private $sem_id;

	public function __construct($key="",$size=0,$perm=0666) {

		if ($key=="")
			$key=__FILE__;

		// default 16KB size shared memory
		if ($size==0)
			$size=1024*16;

		$this->shm_key=$key;

		$this->shm_key=ftok($key,'N');
		$this->shm_id=@shm_attach($this->shm_key,$size,$perm);

		if ( empty($this->shm_id) )  {
			throw new Exception("shared memory allocation failed");
		}

		$this->sem_id=@sem_get($this->shm_key,1,0666,true);

		if ( empty($this->sem_id) ) {
			throw new Exception("sem_get failed");
		}
	}

	public function lock() {
		if ( !sem_acquire($this->sem_id) ) {
			throw new Exception("lock failed");
		}
	}

	public function unlock() {
		if (! @sem_release($this->sem_id) ) {
			throw new Exception("unlock failed");
		}
	}

	public function set($key,$value) {

		$this->lock();
		$res=@shm_get_var($this->shm_id,$this->var_key);
		$this->unlock();

		if ($res===FALSE)
			$res=array();

		$res[$key]=$value;

		if (!  shm_put_var($this->shm_id,$this->var_key,$res) ) {
			throw new Exception("shm_put_var failed");
		}

	}

	public function get($key) {
		$res=@shm_get_var($this->shm_id,$this->var_key);

		if ($res===false) {
			echo "warn array empty\n";
			return false;
		}

		return @$res[$key];
	}

	public function incr($key,$increment=1) {
		$this->lock();

		$res=@shm_get_var($this->shm_id,$this->var_key);

		if ($res===FALSE)
			$res=array();

		if ( empty($res[$key]) )
			$res[$key]=0;

		$res[$key]+=$increment;

		if (!  shm_put_var($this->shm_id,$this->var_key,$res) ) {
			$this->unlock();
			throw new Exception("shm_put_var failed");
		}

		$this->unlock();

		return $res[$key];

	}

	public function  __destruct() {
		$this->unlock();
	}

}

Using it is as simple as the following:

include_once("sharedMemoryStore.php");

$s=new sharedMemoryStore("some-identifier");

$s->set("foo","bar");   // set key "foo" to value "bar"
echo "getting foo:";
echo $s->get("foo");        // get the value of foo
echo "\n";

$s->incr("counter",2); //incremement "counter" with 2
$s->incr("counter"); //incremement "counter" with 1

echo "counter is now: " . $s->get("counter") . "\n";

Now, if you fire up multiple processes on the same machine, they can all share the same counters, or hashtable. The shared memory segment is appropriately locked, so that only a single process can access the variables at a time. incr() works atomically, as expected, so that multiple process can increment a counter without treading on each others’ toes.

The size of the shared memory segment is by default a bit small (16kb), because most Linux distributions don’t have decent shared memory defaults. If you need to increase the size, check the appropriate sysctl.conf setting for your distribution, and change the constructor to something like

$s=new sharedMemoryStore("some-identifier",1024*1024; //1 meg shared segment

I went from about 300 requests per second to 1500 requests per second on a low-spec virtual machine by simply dropping in the shared memory storage class.

I’ve been working with OpenRADIUS for nearly four  years now. I build high-availability and carrier grade RADIUS stuff with it. And I can only endow respect, and “I’m not worthy’s” unto it’s creator: Emile van Bergen,

I chose OpenRADIUS when Neology started, for a number of reasons:

1.  It extends like Apache. With FastCGI, except better.
2. The behaviour language allows me to do pretty much anything. It’s like Assembler.
3.  Emile took the UNIX paradigm of programming, and applied it to RADIUS.
4. The proxy code is phenomenal.
5. Dictionary support is excellent, and adding new ones, a cinch.

OpenRADIUS is like the GAS of RADIUS. I won’t recommend it to any newbie in trying to get things going, but as I’ve  had to add more features, and plumb more cleverness into OpenRADIUS I keep looking back at the days when I researched things like FreeRADIUS, and just cannot believe the quality of OpenRADIUS from as produced (largely) by single developer. The website, isn’t too snazzy, but the documentation is brilliant. I think Emile is so tied up with making a living that the project sometimes suffers from not enough marketing and “bling”, but then again, I prefer technical information above flashy logos and useless wiki’s full of cookbooks.

Ik til m’n hoed voor u, Heer van Bergen.