While working on a box, I noticed that the aus-snmpd binary wouldn't stay running on some machines. This is problematic, because the aus-snmpd binary is used by snmpd to remotely poll the raid card for failed drives.

I had previously created an init script for it. Just in case anyone wants it, here. It should work on RHEL/CentOS/Debian/Ubuntu.

--
I ran the binary with strace, and then poked through the log to discover:

10497 select(8, [3 5 7], NULL, NULL, {0, 999999}) = 1 (in [7], left {0, 999999})

10497 getsockname(7, {sa_family=AF_FILE, path=@""}, [9579780311546331138]) = 0

10497 recvfrom(7, "\1\22\0\0\221\1\0\0\0\0\0\0\263\316\330x \0\0\0F\323\250\25\0\0\0\0\5\0\0\0"..., 65536, 0, NULL, NULL) = 52

10497 times({tms_utime=2, tms_stime=0, tms_cutime=0, tms_cstime=0}) = 938907755

10497 times({tms_utime=2, tms_stime=0, tms_cutime=0, tms_cstime=0}) = 938907755

10497 semget(IPC_PRIVATE, 1, IPC_CREAT|0) = -1 ENOSPC (No space left on device) <--------- Right there.

10497 exit_group(0)

So the solution here is to clear them.

for i in `ipcs -s | awk {'print $2'} | egrep '[[:digit:]]{3,6}'`; do ipcrm -s $i ; done

Then restart aus-snmpd, and it should stay started this time.

I had always figured this would be a thing eventually. Surprise, it finally is. You can open a text document and edit it in (pseudo)real-time while everyone else(over the interwebs) edits the doc too.

It has a chat system built in and uses highlighting to distinguish the typists apart. The chat system resembles gtalk's gmail interface, minus the awesome pop-out feature. You have some basic font features, bold, italic, etc... For what it is, the UI is pretty nice.
-
The install is your typical build from source... ./conf...make;...install...cake. My only complaints are the packaging, and lack of init script. The installer itself, beautiful. But why give me a tarball? Maybe i'm just being lazy, but everyone provides a .rpm or a .deb. More importantly, the init script!
-
I ended up just running it in a screen session for now. I will make an init script for it at some point.

It won't be something I maintain, so this link probably won't work forever. But etherpad lite

So I changed things up a bit. Once I had everything setup... I realized I hated the 2924s and these boxes are so underpowered for what I am trying to do. If I am going to build my ideal network, I am going to do it right!

I need to get rid of these two 1u boxes, and start over. Ideally, I need something quiet, and powerful. I haven't done much research on that one yet. I am going to start from the top down.

The first part I am going to solve is the switch. Something about bottlenecking my internal network at 100mbit because of the 2924s was aggravating me. So, as an alternative, I am going to go with a Foundry EdgeIron. I really hope, just like ServerIrons, these are forgotten about pieces of gold(I am a little bias toward ServerIrons, can you tell?). It was $125 with shipping. Whereas a 24 port Cisco 2960G used would run me ~$800. Why does it cost so much to be a computer geek?!

The new switch(EdgeIron) has 24 gig ports. It should be here in the next few days. Looking at the manual: EdgeIron_User.pdf its CLI is very similar to Cisco IOS. Maybe I should pursue that FNCNE BCNE after all.

I knew this day would come. I found another ServerIronXL with fiber ports. I'm going to complete my setup with ethernet, but I will probably move over to fiber once I stumble upon a fiber switch.

Setting it up isn't hard. The Brocade manual on this is pretty straight forward SLBGuide HA.

--

One of the things I love about the Foundry/Brocade ServerIron is that they are priced so cheap when you get them second hand. With the price as low as it is, it is totally worth paying for two, to get the additional HA functionality.

Unfortunately this is going to utilize another 2U in the rack. That means I am out of space. Hmm, do I get another 11U rack, or a new rack all together? This was a damn good rack for its price.

As much fun as this project is, its time to clean up my toys.



So I needed to rack the three 2924s. I ordered a few brackets from cablesandkits.com.



I'm going to stick with the Agg switch idea, and the redundant uplinks for the servers. So I racked one of the switches on the bottom. The rack has a lean that ultimately makes the bottom 1-2U useless. But its perfect for a rackmounted power strip!



I know, the cables look terrible. That is going to change soon!

Lets go ahead and configure out Load Balancer. I wanted to use the fiber ports, but unfortunately I don't have a fiber switch...yet.

This ServerIron XL is used, and still had its old config. on it. This means, I would need to reset the password. The way to do this has already been covered on this guy's site.

I did a little restructuring.

LB[eth1] -> SW[fa0/15] VLAN2
FW[eth0/6] -> SW[fa0/16] VLAN1 outside 192.168.1.1/24
FW[eth0/7] -> SW[fa0/17] VLAN2 inside 172.16.0.1/24

-

Lets go ahead and configure the Switch port for the Load Balancer.

B1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
B1(config)#int Fa0/15
B1(config-if)#switchport access vlan 2
B1(config-if)#desc LBp1 inside
B1(config-if)#exit
B1(config)#exit
B1#sh run int fa0/15
Building configuration...

Current configuration:
!
interface FastEthernet0/15
description LBp1 inside
switchport access vlan 2
end

B1#

--

Now lets take a look at the LBs eth1.

lb1#conf t
lb1(config)#sh int brief
Port Link State Dupl Speed Trunk Tag Priori MAC Name
1 Up Forward Full 100M None No Normal 00e0.5206.2360
2 Down None None None None No Normal 00e0.5206.2361
3 Down None None None None No Normal 00e0.5206.2362
4 Down None None None None No Normal 00e0.5206.2363
5 Down None None None None No Normal 00e0.5206.2364
6 Down None None None None No Normal 00e0.5206.2365
7 Down None None None None No Normal 00e0.5206.2366
8 Down None None None None No Normal 00e0.5206.2367
9 Down None None None None No Normal 00e0.5206.2368
10 Down None None None None No Normal 00e0.5206.2369
lb1(config)#

We should be good admins, and name things appropriately. We also need to create a VLAN, and assign an Ethernet port to it. Since we are only dealing with one VLAN, there is no reason to tag traffic.

lb1(config)#vlan 2 name inside by port
lb1(config-vlan-2)#untagged ethe 1
added untagged port ethe 1 to port-vlan 2.
lb1(config-vlan-2)#no span
lb1(config-vlan-2)#exit
lb1(config)#sh vlan 2
Total PORT-VLAN entries: 2
Maximum PORT-VLAN entries: 9
PORT-VLAN 2, Name inside, Priority Normal, Spanning tree Off
Untagged Ports: 1
Tagged Ports: None
Uplink Ports: None

lb1(config)#

Next, lets go ahead and give our Load Balancer an IP, so we can get off the console. I also took the opportunity to add a resolver.

lb1(config)#ip address 172.16.0.254/24
lb1(config)#ip default-gateway 172.16.0.1
lb1(config)#ip dns server-address 8.8.8.8
lb1(config)#

Next, clear any old crypto keys, and/or generate a new one.
crypto key zeroize -- will get rid of the old RSA key.
crypto key generate rsa -- will create a new RSA key.

At this point, if everything went as planned, you should be able to SSH in.
[root@optimus ~]# ifconfig | grep inet\ addr
inet addr:172.16.0.10 Bcast:172.16.0.255 Mask:255.255.255.0
inet addr:127.0.0.1 Mask:255.0.0.0
[root@optimus ~]# ssh -l root 172.16.0.254
SSH@lb1>en
User Name:root
Password:
SSH@lb1#

Don't forget to "write mem"!

The Cisco 2924XL is a Layer 2 switch. This means you get one Management VLAN. But what does that MEAN!?

Basically it means you can communicate with the switch on one interface at a time. This has nothing to do with being able to create VLANs. But, that IP you put on the switch, if its in one VLAN, and you have another IP in VLAN2 on the same switch, it won't work on both interfaces at once.

B1#sh run int vlan 1
Building configuration...

Current configuration:
!
interface VLAN1
ip address 192.168.1.202 255.255.255.0
no ip directed-broadcast
ip nat outside
end

B1#sh run int vlan 2
Building configuration...

Current configuration:
!
interface VLAN2
ip address 172.16.0.120 255.255.255.0
no ip directed-broadcast
shutdown
end

B1#

If we were in VLAN2, and trying to telnet to 172.16.0.120, it would fail. Because the switches management ish is in VLAN1.

Does this have anything to do with being able to use separate VLANs on my switch? No. Interface VLAN is basically a routed VLAN.

B1#sh vlan brief | i Fa0/17
2 inside active Fa0/1, Fa0/2, Fa0/17
B1#

For instance, even though the running config says shutdown, if we look at "sh vlan brief", it says active. This is because the interface VLAN2 is shutdown(not active), but VLAN2 is active. We can also pass traffic without a problem over VLAN2.

In short "int vlan" != "vlan". This was a concept I had to learn fairly late in the game. So I figured it was worth mentioning.