We all know where this is headed right?

I start to pull the power cables, and it is cramped, hot, and not easy work.  Not exactly hard, but not fun.  So I get all the spare cables out, but suddenly the site, it is down.

I work with Mark to find out an access switch got unplugged.  I plug it back in, and it refuses to work.  Steal some network serial connections, give Mark remote access, and it comes back.

Site is still wonky…

It seems that an access siwthc in the adjacent rack had been powered off the power strip in the rack I was working on…. the hell you say?!?  Cross-rack power connections are evil. So I plug it back in, and I hop back in IRC.  I own up to my own stupid mistake that took Wikipedia and all its related projects offline for about 20 minutes or so.  The result?

Praise.  Not for admitting it, but for fixing it.  Proof positive that all the wiki-folks are either just plain nice, or insane.

Having worked with them for two years now, insanity recognizes its own, and they are also nice folks ;]

I hired a buddy of mine to do the ‘OMG, Rob is out of town and the servers are down, fix now!’ kind of stuff.  He works in another datacenter, and knows how to handle himself around servers and the like.  He comes downtown today to get his access setup and see the site.

We get all that done just before 4 and get in the elevator.  I need to get my car at the 3rd floor, and he has to go to 1 and go outside for his car.  Elevator gets to floor 3, DING, door slides open 3 inches…and stops.  Closes, opens 3 inches, stops…. wash, rinse, repeat for a few minutes.  We try to pull it open, it locks in place at 3″.  We press the emergency call, can barely hear the person on the speakerphone.  They said they would send someone.

We wait another 15 minutes, and the door stops trying to open, the motor has burned out, and smoke starts filling the car.  AWESOME!  So we call again, and the person who picks up doesnt know where we are, who were are, what town or state we are in, and can barely hear us, and we cannot understand her very well.  Rich decides smoke + elevator = 911 call and calls 911.  We wait….

10 more minutes or so….

Smoke stops pouring into the elevator, the burning smell of electronics is stinkin the place up, but no more smoke, yay?  So we wait…

10 more minutes or so, and we can hear folks outside the elevator yelling to us.  We tell them what floor we are on, they start talking to us.  Then the elevator decides to go to floor 4 (WTF) and we yell and they yell and 10 minutes later (we actually had our phones out timing this insanity) they are trying to pry the doors open with a crowbar or something, because its making marks at the center opening of the door.  They fail…

The car then goes to the first floor, and sits there for 25 minutes.  We get tired of waiting, and both say ‘Hell, lets try to pull it open.’  I stand up, and I am able to do so without Rich helping me.  It seems when they fired off the fireman override to ground all the elevators, it did something to the doors.  We save ourselves and walk into the lobby.  We see to fireman, conversation goes as follows”

Me: Hey, you know that two guys in car 4 called you right?

Fireman: Yea, we are trying to find out why it happened.

Me: Well, no one RESCUED THEM.  WE ARE THOSE GUYS AND HAD TO PRY THE DOOR OPEN OURSELVES.

FM: Looks imcompetant and mutters something about Fire Marshal is handing blah blah blah.

Me: Yea whatever, the point is we were stuck in there.

FM: We are doing a floor by floor check.

Me: Wouldn’t it make sense to rescue the two gents who called you in the first place, who reported at that time that thier car was filling with smoke?

FM: Stupid look.

Me: Fantastic…

Now, I respect that these folks go in and rescue ppl.  I do not like they NEGLECTED TO RESCUE ME AND MY BUDDY RICH.

Times are approx.  We did not write things down in the elevator.  However, we got in the elevator at 3:58 and did not walk outside until 5:04.

• Ubuntu 7.10
• Apache 2.2
• OpenSSL 0.9.8e

You will want both apache and openssl installed. You can do this by:

apt-get install apache2 openss

This walkthrough will NOT walk you through getting apache working, sorry. It will let you generate private keys, the CSR to submit to an SSL signer and/or generate your own SSL signature.

To make it encrypted with a password (This requires user entry upon apache restart):

openssl genrsa -des3 -out domainname.key 1024

To make it without the password:

openssl genrsa -out domainname.key 1024

Now, you need to generate the CSR. If you made the key with a password, be prepared to type it in when prompted. You also get to type in that password whenever you restart apache (good times). When you create the CSR, the stuff you put in here isn’t all that important, EXCEPT for the common name. If the website you are making a ssl certificate for is for say, blog.robhalsell.com, then that is what I would put in for the common name.

openssl req -new -key domainname.key -out domainname.csr

If you want to verify the contents of the CSR:

openssl req -noout -text -in domainname.csr

Now, when the SSL vendor asks for your CSR, just do the following:

cat domainname.csr

Then copy and paste the entirety of the file to the vendor.

Now, perhaps you want to enable HTTPS on your server just for the secure back and forth of your information. That is fine, then you really may not need to pay someone to generate/assign you a SSL Certificate. The drawback of not doing it with a vendor is the browsers that folks use to hit your site may give them all kinds of warnings about the certificate being unverified. If that is not a big deal, and you really only want the SSL enabled to secure the transactions of data (like say folks logging into your blog or whatever) you can sign your own key, and generate your own certificate as follows:

openssl x509 -req -days 365 -in domainname.csr -signkey domainname.key -out domainname.crt

This generates a certificate good for 365 days. You can of course make that longer or shorter.

If you used a vendor, they will email you with your certificate. Make sure you create a server.crt on your server, and paste the contents of the certificate EXACTLY including all the dashes before and after it. If you fail to do this, well, you get to deal with it.

You will want to move your certificate file and your key file into their new homes on your server. Some folks choose to put these in the same file, I am not one of those folks. With my install of openssl, it made /etc/ssl. In /etc/ssl/ there is certs/ and private/.

Put your domainname.crt in the /etc/ssl/certs/ directory. Put your private key file (domainname.key) in /etc/ssl/private/. Also, at this juncture, BACKUP YOUR PRIVATE KEY AND YOUR CERTIFICATE FILE OR YOU WILL BE SORRY WHEN YOUR HDD CRASHES IN THE SERVER. Seriously, back that crap up. If you signed your own keys, its not that big a deal, you can just reissue them. If you paid someone to sign them, well, they may charge you (you can get reissue insurance with many vendors, which is a nice way of saying ‘I want to pay you for this because I am too foolish to backup my own crap.’)

Once you move the files, you get to play in the apache virtual host files. Again, this is not about how to setup Apache, just add HTTPS to it.

First, enable the ssl.conf and ssl.load. Go into your /etc/apache2/mods-enabled. Check and see if they are there (hint, the command is ls). If they are not, do the following (which worked on my server, your results may vary):

cd /etc/apache2/mods-enabled
ln -s ../mods-available/ssl.conf ssl.conf
ln -s ../mods-available/ssl.load ssl.load

That tells apache to load the SSL module when you restart it. (I am honestly not sure if you just need the .load or the .load and .conf. I put both, but I am wrong an awful lot. ;)

Now, time to edit /etc/apache/ports.conf. Make sure it has Listen 443. You can be fancy and have it only enable port 443 if you have the ssl module loaded. Here is my file:

Listen 80

<IfModule mod_ssl.c>
Listen 443
</IfModule>

Of course, by ‘my file’ I mean the default one that was installed when I installed Apache.

Now, time to edit your Virtual Hosts files, you will want to add in all the virtual host goodness you already have, plus the following:

<VirtualHost www.yourdomain.com:443>
All the crap that was in my original VH file and I put the SSL crap after the DocumentRoot declaration.
SSLEngine on
SSLCertificateFile /etc/ssl/certs/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key
All the other crap in my original VH file following the DocumentRoot declaration.
</VirtualHost>

Now, just restart Apache. You can do this with whatever means you like:

apache2ctl restart

-or-

/etc/init.d/apache2 restart

-or-

Whatever, reboot your damned server if you are too lazy to use the two above examples.

Now, when you go to https://domainname.whatever, you will load up with HTTPS/SSL, yay! This will ensure security and whatnot for password and other client/server transactions.

If this even helps one damned person setup HTTPS on their server, then it was worth it. All the instructions I see out there tend to overlook something, and mine is no exception to that fact. I am quite certain that my instructions will not work for everyone. Like so many other things on the web, its about collating the data and using what you find.

I am not sure if they are going to push this patch in to release versions in the future or what, still finding that out. However, the fact that I found an issue and it was resolved in days is in itself fantastic.

I have the patch installed and working on my DEV server, but that isn’t going to see if it fixed the issue, just ensures the patch won’t kill my servers is all. Will roll out the fix, if all testing goes well, this Wednesday morning.

So in my book, Jay is fantastic.

(Yes, you can see I took out my extension and my cell number, I just do not want to push those out across the entire internet.)

Example, I just put in a backup server, I wanted to call it backup1, but since we will not have a ton of these generic backup servers (all databases are clustered, these backup servers are for corporate and non-wiki data only), Mark suggested I name it something interesting. Of course, everyone else likes interesting, so fine, fine, I called it Tridge. Rsync was invented by Andrew Tridgell, whose nickname was Tridge. There, its cute and creative.

However, what bugs me even more are the release names for Ubuntu. I love Ubuntu, I think the concept and execution of it are fantastic. That being said, I dislike the code names for the projects. You can see them spelled out here. The numerical names are for the month and year they are released. 8.04 means it was released in April of 2008. That makes sense! Now everyone calls it Hardy, not 8.04 =[

My even complaining about this is very much an example of ’tilting at windmills.’ I guess I will just polish my armor for my date with Dulcinea later. (Two points for anyone who gets the above reference.)

However, during the move, I also changed our account name to something that isnt my name.

I have the ‘friend’ name of teliax, because that is the service name. (most of the iax.conf file was removed before slapping it on here, no reason to share irrelevant data.)

***from iax.conf***
register => myusername:mypassword@oneofteliax’sserverips

[teliax]
context=incoming
type=friend
host=oneofteliax’sserverips
username=myusername
secret=mypassword

Now when I got a call, I would get this:

[May 29 13:49:52] NOTICE[10493]: chan_iax2.c:7383 socket_process: Rejected connect attempt from 74.201.8.23, who was trying to reach ‘4154130677@’

Great, now when it gets a call, it has no idea what context to put it in, but why?

I changed the [teliax] to [myusername] and BAM, it tosses it into incoming like I want. Of course, incoming has just s extensions… So I had to put in a cheat:

[incoming]
exten => 4154130677,1,Goto(s,1)
exten => s,1,Answer()

So now it puts the calls through correctly in to our menu system. The annoying part of this is my username is the damned entry name in iax.conf. This is not exactly clean to me because multiple iax connections could conceivably use the same username.

Now that I can see the issue, and how I fixed it, I should be able to copy all my settings to my development server and recreate the issue. It may be as simple as splitting them up into user and peer entries, but I am not sure yet. I do not want to tinker with it on my production server, as that is the main phone system for the office. Having phones working at the office is kinda a big deal.

For the wiki-holics:

The 415 number above is NOT the office’s main phone number, its a secondary number used in a complicated means of forwarding things around so my phone system works. You can call it if you really want, but it will not always work. Within a few months, I will have resolved our forwarding woes, and only the main phone number listed on the website will work.

Exim is splitting it into a multi-part message, and then pushing out the first part and nothing else? These files should not be hitting above the 50 MB attachment limit, but I suppose they are. So now I need to take up some more of Mark’s time to help me with this, as I am a damned Exim newb and he is our mail administrator.

UPDATE:

Mark says Exim is not the culprit, Asterisk is. As Exim is his baby, I am inclined never to argue with Mark about Exim.

So now I am following up with my pal Jared over at Digium. Jared has not seen this, but it may have something to do with the corrupted callerid information. He is going to do some testing on his end and see what arises.

MORE UPDATE:

I finally dug in and compared the emails with the call records, and was able to find the thing that was common in all of them. I then submitted the bug (linked here) to Digium’s Bug Tracker.

Well, I would be pissed off there was a fire, and I would also be displeased that when I called 911, that I may not get a local dispatch and/or they will not know my address until I tell it to them! It seems the e911 feature is not working correctly on parsing some odd addresses. Granted, the address in question is ### Streetname St., so that seems pretty standard.

However, as the little missing green arrow tells it, it ain’t standard! It seems when I put in the address, it is not parsing it correctly. To give Teliax credit, they do have the programmers of their new web portal working on the issue. The blank space where the green arrow should appear, it mocks me…

So now, I start anew, for the third (or fourth?) time. I should follow this with the promises to update this thing on a regular basis, and so on, and so forth, but damn, we all know there is a good chance I never will.

Thus deploys yet another version of Rob’s blog.