Simple Testing

A while ago I've upgraded my whole development AS infrastructure to 10.1.2.3, after the upgrade SSO stopped working. Since SSO (at least mine) tends to be a bit fragile, I sighed with a familiar feeling that this is going to take some time and started browsing through the logs. Once again Oracle proved that their logs suck and tend to display the same error message for totally different errors. So the logs showed an error that usually accompanies a bad keytab file, since I had some similar issues lately I've decided to re-create the keytab file, but it didn't really help. Apparently, all I had to do is to search the Metalink. 

The real issue is that the jdk version this version(actually not only this one, if I'm not mistaken IDM 10.1.4.0.1 is as well) is shipped with (1.4.2_14) has some error that prevents SSO from working, the solution is a simple one - install a higher version (say 1.4.2_19).

Now, these things get me really frustrated. OK, I get it, your error handling is not the thing you take pride in (sure hope not), but it's not the first time (more examples to come in following posts or you can read this post again [last example]) it seems nobody have really tested the final product. Had somebody taken the final product (with jdk 1.4.2_14) about to be published for everyone to download, installed it and tested it for the very basic functionality this error would've been discovered (and hopefully the product wouldn't be published).

That's the part when I start to imagine the following conversation (a special bonus for whoever discovers the meaning behind the aliases):

M: "Hey, there's a new jdk out! We should ship 10.1.2.3 with it because it's the newest" (and new is good, right?)

S: "But we tested it with 1.4.2_x<14!"

M: "Yeah man, but we didn't really touch something that heavily depends on jdk specifics, it's a minor version anyway."

S: "You know what, you're totally right, let's do it! It's not like we've ever shipped any totally-unworking piece of code before."

Really, with the simple applications we have in my company we try to test them thoroughly and even though not always successful try to enforce different rules before deploying, things like a certain period of time in which the application has to work on a test environment without code modifications before moving to production, a clean testing environment for relatively big installations, etc.. 

So how come Oracle manages to ship a totally not working version of a product?

Noise

Well, I haven't blogged for a while, you can probably blame halo 3 for that as well as some other XBOX games. Anyway this post is not going to be too professional as I just want to raise an issue that deeply bothers me.

Lately I have been to several meetings in different companies, at most times I haven't been too satisfied with the hosting. How come? 

Well, there's this issue with cookies and other snacks placed on the table, they tend to be noisy - I mean every bite you take makes a sound so loud it reminds me of the reasons I haven't blogged for a while, if you know what I mean. And so instead of enjoying a snack I end up trying to silence the breaking sounds - can't people provide snacks that behave?

In one company they gave me a drink in a glass bottle, the problem is that the table in the conference room was made of glass as well, so putting the bottle back on the table after drinking without generating a loud response required great skill - I kinda started imagining me as a Terminator measuring the distance to the table and the amount of force needed to put the bottle back silently.

Please people, think of me next time...

Digging

One of the abilities a sysadmin must have is the ability to dig. What do I mean when I say "dig"? Well, I mean that many times you face a problem that seems to be rooted deeply inside the system, and a sysadmin has to know how to solve these problems. The first thing I personally do most of the times is to say "OK, that's as far as I'm going, time for official support", but as I wrote before, this usually ends up with me de-compiling Oracle's code.

Time for an example.

I've recently been trying to utilize Oracle's ESM (Enterprise Security Manager), since the system it's intended for requires batch actions I used the command line tool. Guess what, when I use one of the commands the GUI stops showing associated roles for the related Enterprise Role, since it was obviously a coding bug, I've logged an SR thinking to myself that this bug, if reproducible (and it is) means no one have ever tested this functionality. About a month later I was notified that a fix was created and they're testing it. Two days later - bad news, the fix did not pass testing. And this is actually the current status (a few months have passed by now).

Last week I got tired of this whole thing  so I've started digging for the source of the problem. Apparently, when you use the command line, the record generated in the OID has a case mistake. So my next move was to de-compile Oracle's code and fix it, and voila, it works!

When I told the support analyst about my fix, he said he knows the solution is easy but the problem is that they have some incompatibility issue with different database versions. I really don't care if it's reasonable for such a small fix to be taking such a long time to implement, I'm just happy I know how to dig.

Life-Work Balance

It doesn't matter how intensive your work is, how time consuming your studies are and how tired you get home after all of them. You should always make time for some other activity, it can be sports, some artistic hobby or whatever, as long as you enjoy it. Another important thing about this hobby is that it must be something you do constantly and don't skip no matter what. It might sound complicated but it's actually as simple as forcing it into your schedule and not giving up on it (and to yourself).

For me, practicing Judo is a great way to get my mind set on something other than work and everyday thoughts, it makes me concentrate on an entirely different thing. One of the things I like about it is that the great physical effort you have to put into a training session is a huge contrast to my work, that involves mostly sitting in front of a computer in different angles.

Yes, it can get tedious, making progress is hard and there are those days when I have to struggle myself not to give up on the training session. But the fact is that most of the time and especially after showing up for a session after missing a few I have real, genuine fun and that's what it's all about.

Traveling

I haven't blogged in a month, and for a good reason too. I was traveling Ireland for two and a half weeks and after physically landing it took some time to mentally land and even more time to "catch up with life" - organize my photos from Ireland, watch all the episodes I've missed while being abroad, etc..

This trip is obviously something I should write about, my trip companions actually offered all kind of ideas during the trip, but I don't really know what to write. As I write those lines I don't know yet what I will write about. Really. I guess I'll just write some outline of the trip and interesting highlights for those of you who plan to visit Ireland as well.

Part I - Shortened version of the Wicklow Way

Since we wanted to make the most of the time we had and the last part seemed a bit boring, we decided to start from Dublin and walk only for five days - up to Moyne/Tinahely and not up to Clonegal. 

The walk was really great. Especially since we had no rain at all during walking (and in general great weather), actually we had lots of luck since everyone we talked with before the trip told us we'd be totally soaked inside out, in reality the massive coat I brought ended up as just a burden, the only times I took it out of my bag was when I moved it to another bag. We had some rain during the first two evenings and some rain on the last two days in Ireland - and that's it. Oh, and one evening when one of the guys hung his clothes to dry outside, but that wasn't me so it doesn't count. All in all this absence of rain is another proof to my belief that I have tons of excess luck. Anyway, words (except maybe "Green") won't really describe our walk so you can either check out the photos or walk the Wicklow Way yourself (recommended option), and we've learned a few lessons as well:

Tip 1: Public transportation in Ireland sucks:

a. It's expansive - buses are expansive, taxis are expansive and trains I prefer to forget (31.5 euro for a train from Galway to Dublin).

b. It's hard to find. When planning to quit the Wicklow Way early we didn't consider the fact that we won't be able to find a bus and the "taxi" we eventually used was an elderly guy with a van (that robbed us nonetheless).

So what do you do? Whenever possible just rent a car, this method has it's own limitations (drop-off points and the like) but it's much more convenient and cheaper.

Tip 2: Don't carry around your big bags. The places you stay at, will be glad to transfer your belongings to your next stop for a fee (usually 20 euros), but it's worth it. And if you picked Ireland you can probably afford it.

Funny 1: We're lame. During the walk we've met many people doing the Wicklow Way. Many of those people aged more than twice our own age and they didn't even bother listening to my second tip... embarrassment indeed.

Part II - The South

Next we rented a car and visited:

a. Around Cork: 

Blarney Castle - a nice castle which has the "Stone of Eloquence" on the top floor, kissing it is not a simple task, so unlike other visitors we decided to pass.

Fota Wildlife Park - a very nice wildlife park - unlike a zoo, animals are not behind bars and mostly (not the cheetahs though) are separated from humans only by water or modest wire fences. 

b. Around Killarney:

A whole day of cycling in the Killarney National Park - Ross Castle, the Gap of Dunloe andMuckross House, including sailing through the three lakes by boat. 

This little adventure took some recovering from.

Part III - The West

In this part we dropped of the car and stayed for three days in Doolin - the place where everyone owns a B&B (at least one).

a. Walked the section of the Burren Way from it's beginning to Doolin (we took a bus to the beginning) through the magnificent Cliffs of Moher.

Tip 3: The Burren Way. Looking now (we didn't have a map as we did the walk) at maps I have, it seems there are at least two versions of the Burren Way - one that starts in Lahinch and simply skips the Cliffs of Moher and one that starts at Lisacnnor and goes through the cliffs. We tried to do the second version but it was very poorly signed (unlike the Wicklow Way), luckily we realized going down at the cliffs didn't make sense.

b. Took a ferry to the largest of the Aran Islands - Inishmor - and cycled through it (this time it was much easier).

Tip 4: Unless you stay for the night you have only about three hours to visit the island before the return ferry arrives.

Tip 5: There's an option to take ferries for the islands from Galway as well, an option worth exploring.

c. Spent about half a day in Galway.

Part IV - Northern Ireland and around

After a brief stop in Dublin to celebrate Rosh-HaShana and rent a car we headed up north.

a. On the way north we visited the megalithic tombs at Bru na Boinne.

b. We stayed at Carlingford near the invisible border and did a short (about 11 km) walking route - poor signing again.

c. Drove to the very north of Northern Ireland and been to the Giant's Causeway. 

Funny 2: When we finished with the Giant's Causeway it was already dark, we were hungry and we had a long long drive back to Carlingford so we entered a nearby hotel to eat. I don't know why but we were total strangers there - everyone looked at us,  they didn't know how to charge us for the beer and asked us to another room to serve us dinner, strange...

Part V - Dublin

We had about two days in Dublin - one before going up north and the last day, can't say the city is too exciting, the Guinness Storehouse is nice but unless you're a Guinness fanatic it's not that much of a thrill. Also there are some nice museums.

That's all folks.

Of course, if anyone is really planning on visiting Ireland, I'll be more than happy to answer any questions.

Buffering...

It's not really clear what I'm doing writing this post when I'm leaving for Ireland in an hour and a half, but I guess I have a buffer to waste...

Anyway, this post is about the single most important part of any IT (any?) project - the BUFFER. A big enough buffer should always be taken for whatever procedure you're planning - we usually take a 30% buffer. 

I'm not talking only about an installation process, but also about things like giving an answer to your manager as to when you'll have a new product checked or a patch installed for the first time on a development environment.  I'd prefer to get a raised eyebrow and maybe surprise my manager later to working under unneccesary pressure and making up execuses. Same goes for actual installation processes, I'd prefer announcing a long downtime and getting complaints from everyone to trying to decide in a rush if I should rollback now or hope I'm lucky enough to shrink an hour's work into fifteen minutes.

Well, that was my little - written in blood - piece of advice before I drown myself in beer, whiskey and god knows what else...

Ignoring My Issues

A big portion of being a sysadmin is solving problems. One of the first changes I've noticed when I stopped being a sysadmin is the fact that I suddenly have (much) more time on my hands, time previously consumed by solving - usually  minor - issues. But I think a sysadmin's job is also to know when to walk away from solving an issue, let me give you a couple examples:

• There's a client side issue with the EBS system I used to administrate, I was never able to reproduce it at will or find it's cause, but I do know that reseting the Windows profile for the client solves the issue every time. So that was my policy, I've decided that if the workaround is so simple and finding a real solution is so complicated (believe me, I've tried), I'd rather just ignore the problem.
• Last week the new EBS sysadmin was trying to solve an issue with one of the EBS related custom applications we have, it has suddenly stopped working without any apparent reason. This application resides on an Oracle Application Server so he was going over the logs trying to find the problem. When I saw it's taking too much time I came over and just restarted the Process Manager and voilla, it works. Talking to him afterwards he said he was aware of this solution but as it was past working hours he thought he'd better explore the issue - well he had a point. But my point is that an issue that happens twice a year or so and that can be simply "killed" isn't worth wasting your time on even if "killing" means affecting some other applications running from the same AS as well.

So how do you know when to ignore a problem? There's no real answer for that but you should probably consider the following aspects: Is there a workaround? How bad is it? What's the frequency? How long to work out a solution?