Having read some of Joe’s views on what a network should look like, and stumbling across “no routing, no PPPoE lameness” I felt that I had to re-examine whether PPPoE is lame, and whether PPP itsself, is dead. It also made me thought more in-depth about my vision of a modern open access network. Back when Neology did it’s OA network in Tshwane, we had a LOAD of technical issues and visions and some really crazy ideas about how to get it done.

Reality, however came when we started implementing.

Joe’s post and architecture is noble. The devil, is as always in the details.

Layer2!=Ethernet

I can’t find fault with Joe’s Layer2 design. As far as it pertains to Metro-E.

Ethernet is simply the most ubiquitous protocol around, and regardless of whether the backhaul switching  incarnation is PBT, or QinQ or whatever collision domain limiting architecture the next vendor may invent, the bottom line around Joe’s Open Access network remains Ethernet.

Everyone’s got it.

How Layer2 is managed in the collision domain, and how Ethernet is trunked back to the  ISP of your choice is largely semantics and arguments around Layer2 backhaul implementations. Back when we did Tshwane, none of these technologies or standard really existed, so we simply adopted a “make things so” approach.

Limiting’s ones scope to Ethernet as Layer2 however is just not entirely viable if you’re considering to be an Open Access network operator.

Ethernet isn’t everything. The fact that I’m still maintaining one of the most feature-rich l2tpns fork’s thats now been IOT’d on everything from 3GPP PDSN’s, IPWireless INC’s  to Huawei WiMAX ASN Gateways, and GGSN’s  is a tribute to Neology  understanding of Layer2 technologies.

We’ve had do things with what we had available, what customers demand, and that has allowed us a great understanding of Layer2 access… Ethernet is but a single aspect of layer 2 access…

“ISP Preselect” is and always has been the core issue as it should be for any Open Access network that’s layered around Access, Management, and Services.

Layer2 Authentication

Joe mentioned that “To get on to the switch fabric beyond my block I use something like dot1x”.

It would be nice, in fact if 802.1x was the defacto mechanism EVERYWHERE for getting your Ethernet layer access, even if it is just “on your own block”. That would allow you to switch ISP’s at a moments notice without waiting for some “NOC Engineer” to “reconfigure” your port on the switch to now plug into ISP(a). It makes absolute sense.

Change your 802.1x credentials and BOOM, you are now trunked into ISP(b) network, rather than ISP(a)’s network.

This is generally achieved by simply slapping a few attributes onto the RADIUS  EAP-TLS/TTLS reply, and  is something that most metro switches these days  understand.

The AAA support in 802.1x means that your desktop, or notebook can  simply supply a certificate, which (gasp) will probably require  a username and password to validate the certficate. EAP-TTLS will simply require a username and password if you’ve got the AAA server cert signed bya part of your CA chain.

All of this  would allow the  Open Access network  to connect you to the service provider of your choice, via VLAN or _insert_funky_metro_backhaul_protocol_here by simpy changing your 802.1x credentials.

It also (quite neatly) allows the OpenAccess network provider and ISP to comply with some of South Africa’s niggly regulations such as RICA, that requires the loop provider, and ISP to positively identify their customer.

The problematic part itself is 802.1x and how it shits on the user experience.

The EAP Challenge

802.1x generally uses some kind of EAP based authentication protocol. So, which EAP would you like to use today ?

PEAP, EAP-TLS, EAP-MD5, EAP-TTLS, EAP-(INSERT_TLA_HERE)?

EAP, in it’s bountiful incarnations represents a number of  challenges:

Firstly, that of client-side support: Yes, Windows has it, MacOS (maybe) has it, Linux has it. If you install the right package, and twiddle the right knobs. Good luck if you’re running something odd.

Second, is that of a decent trusted PKM infrastructure, which (I’m assuming) is partly what has got Joe so excited.

TrustFabric could play a pivotal role in providing PKM infrastructure services for something such as an 8021.x based Open Access network. There’s some nice dollars to be made in that. However, people like Verisign, Thawte and others have got SAAS PKM infrastructure available as well, and have been playing that game for a long time. Even “open” platforms such as OpenID has made movements in the PKM direction.  It’s a case of “who can issue certs the easiest”.

The final and biggest challenge behind EAP and ANY kind of PKM authentication remains “getting that damned certificate installed”, and working,  and presenting a useful interface to a customer.

In other words,  something that mom and pop can use on their newly acquired Asus EE-PC and MacBook.

Good luck in trying to get people to right click on .pem, .der and other paraphernalia required to get a certificate installed on Windows.

Your best bet, unfortunately  is some kind of “client application” that makes things “easy”, such as Cisco’s or other third party supplicants. Boom, there goes most of your “plug and play” capabilities and “vendor independence”. I’d hate to see an Open Access layer2 network that requires me to authenticate with some muppets’ software that cannot run on Linux, for example.

This is aside from the fact, that the various OS’es built-in 802.1x clients will only support very specific flavours of EAP, and with _very_ specific attributes present in the cert.

The best implementation thus far is EAP-TTLS, which simply requires a username and password, as long as the AAA server is in the client’s trusted chain. This can be easily achieved by having your AAA cert signed by the big boys such as Verisign or Thawte, in a similar fashion to how SSL works.

Unfortunately it’s not supported by Windows, by default, again. EAP is a minefield. If you don’t believe me, just go read this and look at the varied support and implementations.

The bottom line is: Pick your choice of  EAP agony, or go home. Suddenly the Open Access network has become very “closed”, and getting Layer2 access has become more complicated than asking Telkom for a copper pair.

My view on joe’s Layer2: The summary

• 802.1x authenticated Layer2 access is a challenge, not just for the provider, but also the customer.
• Non-802.1x Layer2 access is an customer take-on and ISP switching maintenance nightmare.

It may sound like I’m knocking Joe’s Layer2 vision quite a bit. But I’m not. Really… It’s just the the reality of 802.1x has been so muddied with vendor specifics and “extensible” standards that it’s nigh impossible to implement 802.1x for everyone, which was the entire point behind 802.1x to start with.

The alternative, which is to provide Layer2 services without authentication, or some kind of manual provisioning or vendor-switch specific mechanism is er, well. Vendor specific, or labour intensive. It generally doesn’t allow me to dynamically choose the service provider trunk that I would like my “circuit” to be terminated in.

802.1x should have been the most elegant way to provide dynamic switching between ISP’ Layer2 access to a customer.  Except it isn’t. Maybe it will settle down some in the future. Right now, it’s a frigging nightmare. Personally, I shall wait for the battle of the EAP’s to present a winner.

I fucking hope that winner is EAP-TTLS.

Joe’s Layer3

Joe: “Anybody can sell IP traffic over this switch fabric.”

OK. How do we sell Layer3 over a switch fabric? Well, the selling part should be obvious.

Actually GETTING Layer3 access is another thing. The first step for a customer being getting an IP address.

On a generic (802.1x authenticated) Layer2 Ethernet network the client’s choice of obtaining an IP address is:

1. DHCPv4 the thing a dynamic address.
2. Statically Assign an IPV4 address via DHCP based on the MAC, or customer.
3. Have IPV6 autoconfiguration assign the thing (but hey, isn’t that the same as DHCP?)
4. Use something like PPPoE but Joe doesn’t want this ?

This is all great, and very “plug and play” if we’re talking DHCP, or IPV6 Autoconfig.

Clearly static assignments are a bad thing ™.  So let’s consider that 802.1x actually worked, and I’m trunked back into my ISP’s core, or access devices for customers.

How do I stop a nitwit from assigning an IP to himself that he’s not supposed to have?  The ISP needs a device, to do that. Something that snoops on the DHCP response/reply and makes sure that I cannot (from my MAC) use another IP address.Let’s assume that (most metro switches do this in any case).

The ISP switch, or terminate device now has to filter all traffic for my mac address to be source, and destination correct for the IP address assigned. Fair enough, that’s something that most decent metro switches can do, and the OpenAccess network operator simply trunks traffic from my Metro port to the ISP’s access switch.

Thus, there is traffic inspection required on this  device. It requires ISP switch to do L3 inspection, to enforce the all of these rules. So effectively it’s a layer3 device, or NAS, since it’s probably going to have to generate some RADIUS records as well, to account for traffic. Aside from that, in the case of static IP assignment to a customer, the device will probably also need to advertise my IP’s location to the ISP’s IGP via OSPF, or iBGP.

Dang this access device has become a router rather than a switch. In fact, it’s become a NAS in the traditional sense.

And I’ll bet that suddenly  it has all the scalability problems that a typical “NAS”, PPPoE concentrator, or other “access device” would have, due to the large amount of state, and inspection required.

Now,  let’s take it a step further. The customer doesn’t have a single IP, but perhaps an entire netblock that they want statically routed. They would also like redundancy, and load-balancing across multiple connections!

Oh god. DHCP fails miserably for that. And I’ve yet to see the device that can even implement these sort of things via a simple instruction from a AAA server, and a client device that has support for that kind of functionality.

I could carry on about the problems involved with simply using Ethernet as a DHCP or IP platform but I’d have to write another 50 posts. The bottom line is that the basic Layer3 situation mentioned here has some serious limitations.

We need something else, something that works, something that does more than just handle a single IP, and a single user.

PPP is not dead

PPP is sometimes referred to as “legacy”. In fact, PPP was RFC’ed in 1994.

My definition of “legacy” is “it fucking works”.

Whether it’s PPP over Ethernet, or PPP over GRE (pptp) or PPP over L2TP, or PPP over SoNet is irrelevant.

PPP as a baseline protocol has solved so many networking problems and supports so many features that “modern” ideas of network simply doesn’t support.

Taking enterprise Layer3 access principles and applying them to “circuits” for customers is like trying to take over the entire world with hotspots, walled gardens and DHCP.

PPP is not dead. It supports IPV6, Routed netblocks, bridging (can you say dynamic load-balanced VPLS?) , encrypted authentication, encrypted data transfer, load balancing, bonded, or multiple links. The list just goes on. Every single circuit based networking problem in the past two decades has in some sense been solved with PPP.

Windows, Linux, MacOS and a plethora of other operating systems all support PPPoE as a built-in. Every home “WAN” router I’ve seen supports PPPoE. Very few of them supports 802.1x. Even the most menial Cisco router supports PPPoE.

By the device count on just the above mentioned vendors, PPPoE certainly has a damned good application still in the real world. A market that cannot be ignored, not just from a pervasiveness perspective, but also from a functional perspective.

Layer2 “Pre-select”

The true  challenge for an Open Access network, is to allow the customer to do carrier “pre-select” for his Layer2 Service. This effectively gives them a “circuit” to their provider of choice.

This is what an Open Access network should provide.  ”Layer2 Preselect”. Take my frigging circuit and connect it to the carrier of my choice.

Joe’s case for the 802.1x and L3 DHCP style service is certainly one use-case, but it’s certainly something that is applicable to a general road warrior or home user. It’s based on Ethernet, and IP over Ethernet. That has it’s limitations.

The fact is that PPP is also a Layer2 protocol. PPPoE should be the second defacto Layer2 service provided by any Ethernet-based Open Access Network. To not do so would be suicidal from a business perspective. An OA network design should provide for PPP based “Carrier Preselect” as well as Ethernet based “Carrier Preselect”.

PPP is used on 3G/HSDPA networks, CDMA networks. These are “legacies” that one has to contend with in an Open Access network. Because nice as it may be to dream about, Ethernet isn’t yet everywhere, and there are simply many networking technologies where it doesn’t make sense.

Open Access networks MUST allow for Layer2 preselect. Period. From there on, the implementation should be left to the ISP. The preselect should cater for IP over Ethernet, or IP over PPP. Those are the two most pervasive technologies around.

How to implement PPP over Ethernet?

One of the many possible solutions is fairly simple to implement.

The default “unauthenticated” VLAN for any Metro-E switch that the Open Access network operates trunks through to every ISP that offers services on the OA network. ACL’s are configured to ensure that only PPPoE packets are allowed through this default VLAN, and only to the know MAC’s of the ISP’s PPPoE concentrators. In order to scale it, certain segments of the citywide Metro-E network are trunked back to the provider on different “circuits” or VLANS where they can decide to implement one BIG PPPoE concentrator, or many smaller ones.

Each ISP advertises it’s AC (PPPoE access concentrator) on this default VLAN. A customer wishing to use a specific ISP specifies his username, password, and AC-name, associated with the ISP.

He terminates on the ISP’s PPPoE AC. The ISP pays a per megabit rate for access to this VLAN to the OA operator.

To handle  Joe’s case, the Metro-E switches implement 802.1x. All 802.1X AAA requests are forwarded to the Open Access operators’ AAA servers, which makes a decision to forward the request to the ISP’s AAA based on the outer unencrypted anonymous identity of the Access-Request (which normally still contains the ISP’s domain)

The Open Access operator’s AAA forwards the Access-Request (and ensuing EAP conversations) to the ISP’s AAA, which authenticates the user and reply’s with an Access-Accept to place the user in the ISP’s “service vlan”.

The client DHCP’s and address and get’s his service.

This is but one possible implementation (on the PPPoE side) there are many other options including PPPoE proxy’s, relays, tunneling etc.

QED

An open-access network should provide and support as many industry standards as are possible. Simply providing Ethernet across a nation or Metro is not the entire picture of the solution. One has to consider all the possible use-cases, technologies, supported standards, nice-to-have standards, and their viability.

The fact is that a RADIUS authenticated username and password remains the simplest, most commonly supported standard across a whole range of technologies. How you “make it so” is the differentiator.

I have never had so much fun in multi player online gaming as I’ve had since Left4Dead. Actually I did, in Tribes2. But that’s an era that’s passed.

“The Man Comes Around” song by Cash, is simply incredible. Especially when augmented by a view of the blood of lots and lots and lots of zombies sprayed on your computer screen.

Cash’s lyrics simply makes Armageddon sound like it’s the most romantic and enticing thing on earth. Johnny Cash’s covers are awesome. Honestly, I never thought I’d say that about some guy that lived and made music in an era I didn’t really know about.

“Hurt” is an unknown classic — so underground that you would probably only know about if you wore stockings, tape, and sported black hair in the early nineties, and lived underground and after midnight because it was the “right thing to do”. Or, maybe you bought lots of NiN albums because it was cool.

I did a lot of that — the wearing stockings thing at least… I was also a NightClubOwner. I still meet people in my professional life these days that bring that up and mock me. I wore stockings. I wore duct tape. They considered it weird. meh…. At least I have some stories to tell. They have… Oh, a few years wasted…

I simply look at those days as a seven year hobby. Non-profitable. But extremely enjoyable… It was the most memorable time in my life.

How many 38-year-olds can say they ran an alternative nightclub for 7 years? Fuck — it was my fantasy come true. And I didn’t even have to die the way that Lolly did with all that blood and bullets and shit…

Of course, there’s “Personal Jesus”, by Cash as well:

I can honestly say that by the time Johnny Cash died, he really completed the full, utter and true artistic depth that any artist could strive for.

A true legend. As are the artists he covered. Old learning from the new, and all that jazz…

Get some Fad Gadget.

Tristesse De La Lune, Queen of the Damned. Google it, get the mp3, buy it from wherever.

Then, get out your vampire outfit, dress up  and go and watch ALL of the Underworld movies (again).

If you haven’t, then in which bat cave have you been living?

This article, however was just a priceless win. The subtleties obviously escaped whichever news24 monger was at the helm that day, and simply googled “london 2012 logo”. That, or he/she was actually enough of a subtle bastard to use TheRegister’s version in the hope that nobody would notice…

Either way, it has absolutely made my day.

“We began by responding only to customer demand with our specialized service which derailed our in-house product development. It’s an easy trap to fall in as the money early on is very good. However you will eventually need to invest in your own products and core strengths to build a truly unique and profitable business. We are now highly focused, but I wish we’d done it sooner”

Sam Hutchinson, and Josh Adler of Prefix Technologies

Since apparently it’s fashionable to mention previous business failures in blog posts, I thought I’d mention one of my prior failures here, and then tie it up with focus and customer service.

Prefix Technologies took over Creamer Media’s EngineeringNews, and MiningWeekly after my old IIS/ASP based CMS which served them well for 4 years was becoming simply unmaintainable and unscalable and I started the port to move over to a hybrid Linux/Apache/PHP and Microsoft SQL backend as part of the migration.

It was a trying migration. It worked, but I simply didn’t have enough time and energy to continue the final ports of database migration and then adding the additional features they required. Root cause:  Freeside was  a moonlight engagement for me because we didn’t have enough money as a startup to pay everyone (well, apparently everyone but me) — which meant lack of … yeah, you guessed it — focus.

What was even more trying was that my business partner at the time (Hello Robert Budai) screwed me over no end. For example, not paying the company’s taxes, whilst still happily paying himself.

Entrepreneurial Lessons:

• Never let someone else take care of financial matters and focus purely on technical matters — stay involved.
• Never trust someone else to have done something important (such as paying taxes) based on pure face value — always confirm for yourself.
• Don’t try to start a new business by moonlighting — take the knock in salary, take the pain, but at least you will have focus.

All of the above was pretty much the death knell for Freeside’s original CMS, and the company. I was rather relieved that Prefix could take over Creamer Media. For me — it was time for new business, and a new focus. I resigned from my previous job, and bit the bullet with Neology.

I have to agree with Sam and Josh’s views strongly. FOCUS. Don’t deviate.

The thing is, that sometimes existing customers and prospective new ones will throw things at you that causes your eye to waiver from that ever so elusive thing called “focus”. Sometimes you have to say no, even when everyone around you will say “but they’re willing to pay good money”.

That might work in an environment where you’re selling widgets, but it does not work in a high-stress, creative environment where your employees are the single greatest asset your company has.

Sometimes the customer JUST ISN’T right, dammit!

Even when they want to pay you a lot of money. It could be that their demands will so detract from your core business and focus that it may be more damaging than useful to engage. (Yes,  regardless of the damn money).

Strategy, architecture, wellbeing and not compromising your core values  sometimes DO have to come first. (Yes,  regardless of the damn money).

Continually placating and accommodating unreasonable customer demands detract you from your focus, and strategy and will eventually end up costing you more than they’re bringing you. There is a balance to be had. The problem is that identifying this kind of customer, or situation is sometimes not easy.

I’ve started implementing a fairly basic rule which I use to gauge whether I’m entering into one of these situations. And it regards the damn money.

“If it remotely  starts feeling like I’m going to have to slut myself out just to get the business, it needs some reconsideration”.

qed

I’m enjoying some “boyz”  time with my son, where where generally just do “the dirty jobs we like to”, climb on the roof and fix WiFi gear without Mom complaining,  and then going for a backside dump into the leather couches to  eat some biltong, endure a marathon series of “Brainiac”, and otherwise devour  ”unhealthy” foods, play PS3  games, play bowls  and generally behave in “manly” fashion.

A single-dad/customer of mine told me that being a single parent is not that hard (his being single not by choice). I agree, from a basic logistic point of view that it is entirely do-able. But there are some serious emotional aspects  that I would just not be able to cope with, if I had to be a single, “entirely responsible for all welfare parent”.

Ruben and I are having a nice time, and all,  but we are seriously missing that “other” half that makes our lives complete. It’s the motherly touch. It’s the attention to detail. It’s the “being able to offload responsibility”  to someone other that just yourself, that simply makes being a complete family unit worthwhile.

It might sound basic, but I  miss the other half of me,  that half that makes sure that we eat decent food, the half that makes sure we don’t do stupid things and the half that makes sure we behave responsibly.

The other half of me that will philosophise over a glass of wine.  Ruben misses the mom that tucks him in (even though I generally do it, and she in turn lets me), and tends the sores, the day’s heartaches and the homework, and the things that “need doing”.

Our Daschies miss the invitational slap on the lap, and the coochie coochie coochie, and the attention that I cannot give ‘em.

In the final analysis,  we’d be “OK” without our Mom/Partner/DaschieMinder if some zombies came and ate her brains, or if some absolutely unnatural disaster struck.  We’d “survive”. But honestly,  I can absolutely not fathom the idea of life without her. “OK” is just not “OK”. I cannot replace her.

It  would really, really, really suck. Actually, it would just be downright miserable.

All I can do right now  is to  wait for the next “major release” which is when she comes back on the 14th of Feb. I will be cheering with little flags and ditty’s on the airport apron as the flight lands.

Single Parenthood?

“Single parenthood” is a misnomer. Some people define themselves as “single parents” even thought there is a second parent. That does not necessarily imply  single parenthood. It’s divorcehood, and what it boils down to is contractual differences between two polarized parties.

Divorce or separation with children inevitably results in what I call a continual “merge window”  – and  feeble attempts to “svn up” and merge the “diffs”  of a child from two different sources. The result will always be substandard.  I reckon it’s an extremely difficult  thing for two parents and a child-in-the-middle to attempt to have a “HEAD” version of life…  Of course — it’s not UNdoable. It’s just very difficult.

“svn up -r divorce” is probably the worst thing you could do to the little human that you’ve (against his/her will) brought into this life. But sometimes these are the cards that are dealt, and sometimes this is the way relationships crumble. At least there is still some argument, and at least there is still (generally) two parents involved.

But, absolute, naked, true,  SINGLE parenting, is an absolute loneliness and responsibility that in my view is almost untenable, and takes  a super extraordinary person to deal with… I know a handful of these extraordinary people. People  that against all odds are the people that manage actual, proper, unadulterated, balanced, SINGLE  parenthood.

You are god-like.

Here’s a pip to you. I’ve been going 10 days. I have an end, some love and a little flag in sight.

You don’t.

You are god-like.

The game was called “Legends”, and was a community and fan driven spiritual successor to the Tribes series after Dynamix was closed down by Vivendi Universal games.

Legends, was based on the then Indie-sourced Tribes engine, redubbed Torque, and released by the ex-Dynamix team under the Garagegames banner.

Marc and I had some discussions around the licensing of his songs, but in the end, I couldn’t really offer him anything else than fame and possibly (mis)fortune for having his music embedded in a completely free game.

I’m glad to see that a few years later, his tracks Industry and Oxyacetylene were released as part of the Gran Turismo series of games.

Cubanate’s music simply has the attitude, and gusto for action games, and would be a perfect fit for any future action games. I’ve not heard of Marc, and Cubanate for many years, but I hope that he has found a niche in sound production and making some commercial money off his great musical talents.

In a (probably overinflated, self-gratuitous) way, I hope that I may have had some influence on making him considering the game and electronic media as a sales avenue for his music.

The sad part is that there is actually very little live, or otherwise staged footage of Cubanate left on the internet. I reckon if Cubanate had ridden the Web2.0 wave a little longer like bands such as Sister Machine Gun, and Nine Inch Nails, they could have continued their awesome industrial electronic hit career. Of course, one can never predict or understand the band’s internal politics and cannot question their personal decisions.

I saw Cubanate live. Once. In South Africa, in Hillbrow, in a derelict broken down building that absolutely suited the crass, industrial nature of Cubanate’s music. It was one of the most awesome experiences in my life.

Long live Cubanate, and one of my favourite tracks:

Other Cubanate tracks to “look” for, in order of priority:

• Lord of the Flies
• Human Drum
• Autonomy
• Forceful

In total — 12 Peers  – less than 72 hours. That’s about a peer per 6 hours. In fact, the first 12 hours turned up the most of the local ones.

It was all as simple as finding a contact in the organisation and exchanging peering details and netblocks. Of course, has been settlement free as well, and Neology will continue to do so as long as it has capacity. And if we don’t have capacity we will endeavor to provide more capacity.

Thanks to all the new peers,  you are part of the drive that is going to make INX’es in South Africa a success in the current and future tense.

It makes sense to peer settlement free at JINX. It’s good for the local internet. Except if BGP and route-filters are rocket science, as is generally the excuse toted by the “unmetionables”.

Sanity check:

It’s actually cheaper for me to get transit to the “unpeerables” via Telkom, rather than attempt to negotiate their prohibitive local peering requirements and pricing. Thanks Telkom, you are my friend. True value for money! Oh wait. Doesn’t that defeat the entire point of JINX ?

Here’s the thing — I’m paying for the SAIX local transit, and the “umnetionables” are paying for it too. So, in the end — we all paid for SAIX transit. If we don’t peer via SAIX or JINX then it would have to go SAIX, or international. So — wouldn’t it just make sense to peer at JINX? Yeah. I thought so.

Finally, many thanks to Graham, and Regardt for “making it so”. And thanks for all the V6 work as well. Neology is probably one of the better connected V6 providers at the moment, simply due to our willingness to do V6. We are hoping to establish peering with all the remaining JINX participants at this point. Basic logistics and time-zone issues appear to be the most common issue. Not “peering” agreements.

To the “umentionables” (you know who you are) … Thanks for your “cooperation” and entirely ridiculous peering policies. o_O