linux

 

 Shared Memory replacement for Memcache in PHP

October 6, 2009

speed

I recently (tonight, actually) had a requirement for the usual set() get() and incr() memcache style key-based variable access in PHP (on Linux) without the usual mucking about of installing memcached, and php5-memcached on Debian. I’m busy writing some modules for OpenRADIUS and they need to be able to increment some shared statistics counters, and share other state information between multiple instances of the same module.

I didn’t need cross-machine sharing of the data, and needed a quick (fairly platform independant) way of sharing some variables between processes. Previously, I relied on memcache, but the socket and stream i/o overhead in a high-performance solution is simply too much for the simple sharing of a hashtable I require.

As any good linux programmer will know, shm_* is your friend. SYSV IPC and shared memory is a fairly light, mostly kernel-based implementation mechanism to share memory across Unix processes.

Without further ado, the class:

// This code is public domain
// Original author: Roelf Diedericks (rodent {at} rodent.za(.)net)

class sharedMemoryStore {

	private $shmk_key;
	private $shm_id;
	private $var_key=1;

	private $sem_id;

	public function __construct($key="",$size=0,$perm=0666) {

		if ($key=="")
			$key=__FILE__;

		// default 16KB size shared memory
		if ($size==0)
			$size=1024*16;

		$this->shm_key=$key;

		$this->shm_key=ftok($key,'N');
		$this->shm_id=@shm_attach($this->shm_key,$size,$perm);

		if ( empty($this->shm_id) )  {
			throw new Exception("shared memory allocation failed");
		}

		$this->sem_id=@sem_get($this->shm_key,1,0666,true);

		if ( empty($this->sem_id) ) {
			throw new Exception("sem_get failed");
		}
	}

	public function lock() {
		if ( !sem_acquire($this->sem_id) ) {
			throw new Exception("lock failed");
		}
	}

	public function unlock() {
		if (! @sem_release($this->sem_id) ) {
			throw new Exception("unlock failed");
		}
	}

	public function set($key,$value) {

		$this->lock();
		$res=@shm_get_var($this->shm_id,$this->var_key);
		$this->unlock();

		if ($res===FALSE)
			$res=array();

		$res[$key]=$value;

		if (!  shm_put_var($this->shm_id,$this->var_key,$res) ) {
			throw new Exception("shm_put_var failed");
		}

	}

	public function get($key) {
		$res=@shm_get_var($this->shm_id,$this->var_key);

		if ($res===false) {
			echo "warn array empty\n";
			return false;
		}

		return @$res[$key];
	}

	public function incr($key,$increment=1) {
		$this->lock();

		$res=@shm_get_var($this->shm_id,$this->var_key);

		if ($res===FALSE)
			$res=array();

		if ( empty($res[$key]) )
			$res[$key]=0;

		$res[$key]+=$increment;

		if (!  shm_put_var($this->shm_id,$this->var_key,$res) ) {
			$this->unlock();
			throw new Exception("shm_put_var failed");
		}

		$this->unlock();

		return $res[$key];

	}

	public function  __destruct() {
		$this->unlock();
	}

}

Using it is as simple as the following:

include_once("sharedMemoryStore.php");

$s=new sharedMemoryStore("some-identifier");

$s->set("foo","bar");   // set key "foo" to value "bar"
echo "getting foo:";
echo $s->get("foo");        // get the value of foo
echo "\n";

$s->incr("counter",2); //incremement "counter" with 2
$s->incr("counter"); //incremement "counter" with 1

echo "counter is now: " . $s->get("counter") . "\n";

Now, if you fire up multiple processes on the same machine, they can all share the same counters, or hashtable. The shared memory segment is appropriately locked, so that only a single process can access the variables at a time. incr() works atomically, as expected, so that multiple process can increment a counter without treading on each others’ toes.

The size of the shared memory segment is by default a bit small (16kb), because most Linux distributions don’t have decent shared memory defaults. If you need to increase the size, check the appropriate sysctl.conf setting for your distribution, and change the constructor to something like

$s=new sharedMemoryStore("some-identifier",1024*1024; //1 meg shared segment

I went from about 300 requests per second to 1500 requests per second on a low-spec virtual machine by simply dropping in the shared memory storage class.

Unix Development | 1 comment
1 response to “Shared Memory replacement for Memcache in PHP”
  1. weasel says:

    Brilliant, exactly what I was looking for!!! Thanks.


Tags: , , ,


 

 DefenseTurret – A Tribes2 Anti-Cheat by TheRoDent

December 26, 2007

Well, Tribes2 ended up being pretty cheat-free, but eventually some people came out with some really bad exploits.

DefenseTurret is my project to kerb these cheats. The program uses an executable injection mechanism to seamlessly load into the tribes2.exe Win32 and tribes Linux binaries.

The basis of the system is a “consensus based” cheat detection mechanisms where clients connected to the server check the validity of each others’ playing environments.

It’s been accepted on the European, and American ladders as the de-facto (read ONLY) anticheat program for Tribes2. I released a Win32, and Linux version.

More information can be found at http://rodent.za.net/defenseturret/, or http://dt.triben.de/dt_en.html

Game Development, Reverse Engineering | Leave a comment
Tags: , , , , ,


 

 How to get local routing(BGP) info for South Africa

November 19, 2006

I recently went on a mission to discover, which subnets are “local” to South African networks, whether by ISP peering arrangements, or direct connection. I wanted this information, so that I could setup my home linux router to use Telkom’s ADSL connection for local traffic, and to use Sentech’s MyWireless connection for international traffic. Reason: Sentech pings aren’t really good for local gaming, but international speeds are great.

BGP (Border Gateway Protocol) 

All route information is published, and synchronized between ISP peers, via the BGP protocol, which is a dynamic routing protocol.

Unfortunately, it’s not as simple as install something like Zebra (a routing daemon for linux that does BGP, set it up on a Linux machine, receiving a BGP feend and have it make clever routing decisions.

No ISP will let you connect to their routers’ BGP port. Easily, or without a fight, or without paying them money for transit. This kind of public routing information, is unfortunately only available to the end-user via a series of public route-servers, and there aren’t any that I know that will allow you to receive the feed via BGP either.

So, I looked at alternative methods. I went from writing scripts to dig through the ripe, arin, and radb databases, to turning to lists of IP ranges arrange by geographic location. All the time, using whois queries to resolve the AS (AutonomousSystem) numbers, and then querying them for their official public routes. The problem is, that these routing databases aren’t always up to date, and that it’s quite difficult to figure out which AS numbers are actually local ISPs.

AS Numbers

An AS number is a  unique number, assigned by ARIN, or RIPE, that defines a BGP routing “area” or an ISP. Internet Solutions’ AS number is 3471. To see the details in the registry for an AS, go to http://www.radb.net/cgi-bin/radb/whois.cgi?obj=AS3741

To see the routes published by this AS, go to http://www.radb.net/cgi-bin/radb/whois.cgi?obj=!gAS3741

There is a set of RESERVED AS numbers, similar to “reserved” IP ranges that is supposed to be used for people that don’t have AS’s to obtain BGP information, or used for private or interior routing. Again, good luck in finding someone that’s prepared to configure a feed for you using a private AS, on a dynamic IP such as ADSL.

In the end, Gregory Massel, of http://www.ispmap.org.za/ fame, helped me to get hold of directly accessible BGP route information, courtesy of telnet://route-server.is.co.za, a public service by Internet Solutions. SAIX also runs a route-server at telnet://tpr-route-server.saix.net/
I wrote a small script that would telnet to this router, and dump the BGP routing table. This table contains local subnets, which is exactly what I was after.

From here on, it’s pretty simple to modify the script to add routes on my linux machine for these subnets on a specific interface. The net result in my scenario: ADSL gets used for local traffic, and Sentech for international.

Example script:

#!/usr/bin/perl

use Net::Telnet;

$prompt = '/public-route-server>/';
$server="route-server.is.co.za";

print "Connecting to $servern";
my $session = Net::Telnet->new(Host => $server,Prompt => $prompt,Timeout=>30);
unlink("t.log");
$session->dump_log("t.log");

$session->waitfor($prompt);

#turn off paging
$session->cmd("terminal length 0");

#get list of local routes
print "Retrieving BGP routes\n";
my @output = $session->cmd("show ip bgp\n");
print @output;
print "Route list received\n";
$session->close;
Internet | 4 comments
4 responses to “How to get local routing(BGP) info for South Africa”
  1. rodch says:

    Hi,
    Thanks for this.

    BTW. Some of the \n nl chars in your perl script lost their \.(As seen by me in FFox)

    To reduce the size, I tried running the list through a couple of cidr subnet aggregators, which fail variously.

    I am cisco-illiterate, Any idea what a subnet like 196.3.161.0 , (without mask bits) means? One aggregator tries 196.3.161.0/32 another barfs with “line too long”

    A quick ping scan tells me there are live hosts within the 196.3.161.0/24 space.

    • roelf says:

      Hi Rod. A subnet without a mask means that it’s a “standard” class prefix. E.g. you can assume /24 for the 196.3.161.0 subnet, or /8 for a 21.23.0.0 subnet. It’s a bit schizophrenic I must admit.

  2. xarion says:

    Awesome, I found that the server now reports a: “local-route-server” prompt so for whoever wants to use the script just needs to change $prompt = ‘/public-route-server>/’; to $prompt = ‘/local-route-server>/’; How do I spew out just the address not all the other columns?

  3. Jaco says:

    Mod on Script:
    1) Use Saix server
    2) Display only destination subnet
    3) Add standard prefix where required

    #!/usr/bin/perl

    use Net::Telnet;

    $prompt=’/tpr-route-server>/’;

    $server=”tpr-route-server.saix.net”;

    $saix_cmd=”show ip bgp community 5713:56225\n”;

    print “Connecting to $server\n”;
    my $session = Net::Telnet->new(Host => $server,Prompt => $prompt,Timeout=>30);
    unlink(“t.log”);
    $session->dump_log(“t.log”);
    $session->login(“saix”, “saix”);

    #turn off paging
    $session->cmd(“terminal length 0”);

    #get list of local routes
    print “Retrieving BGP routes\n”;
    my @output = $session->cmd($saix_cmd);
    foreach (@output) {
    @fields = split(/\s+/);
    $fields[1] =~ s/^i//;
    @parts = split(/\./, $fields[1]);
    if ($parts[3] =~ /\//) {
    print $fields[1] . “\n”;
    } elsif ($parts[0] < 128) {
    print $fields[1] . "/8\n";
    } elsif ($parts[0] < 192) {
    print $fields[1] . "/16\n";
    } elsif ($parts[0] close;


Tags: , , ,


 

 MyWireless/IPWireless P1C USB Modem Drivers for Linux

I wrote a Linux device driver for the IPWireless USB modem that has been included since kernel 2.5 

These can now be found at http://www.neology.co.za/opensource/sentech-mywireless-ipwireless-p1c

I also have some old archives, and collections of tools, firwmare and an alternative driver which was written by some guys for Woosh in New Zealand over here.

The device driver was written by painstakingly reverse engineering the Windows driver using USBSnoopy.

Unix Development | 2 comments
2 responses to “MyWireless/IPWireless P1C USB Modem Drivers for Linux”
  1. aaron says:

    hi, i am begineer using ubuntu and i am trying to get it connected to the internet using woosh. i have tried using the commands on http://www.wlug.org.nz/WooshWireless but it still doesn’t work for me. i really need your help. thank you very much.

  2. roelf says:

    Aaron, those instructions appear to be for the Ethernet based IPWireless modem. If you are using the USB based device you may need to alter your /etc/ppp/peers/woosh file to use the TTY.

    You can look at the information included on the ubuntuforums, e.g. http://ubuntuforums.org/archive/index.php/t-969588.html


Tags: , , , ,