Friday, December 28, 2007

Gotta Have Faith

I'm an atheist.
But I really do understand the need to have something powerful and almighty to believe in. Especially since I'm a sysadmin. How come? Well, the way I see it, two of the reasons people invented god are:
1. People need something to depend on in hard times.
2. People need explanations for the unnatural.
Both apply to sysadmins.

I'll elaborate.

1. Counting on Luck
Sometimes life gets tough. Sometimes you have an issue with some installation that you can't solve (and from how things evolve you know the help won't come from support), you're way beyond schedule and more problems keep rushing at you.
In times like this I tend to tell myself, like a good friend of mine phrased a while ago, "In the very very end things will be OK". Actually, now that I read it, it sounds a bit pessimistic but the intention is actually the opposite, the "very very end" doesn't have to sound so fatal, it just means problems will be solved in their own time. Sometimes, one just have to realize that no matter how grim a situation is, the issue WILL be solved, there's really no way you're not going to succeed in upgrading the god-forsaken database! So why worry (too much)?
This approach might make me sound a bit of an indifferent person, but if that what keeps me going after spending a couple of weeks trying to solve an issue and not making any progress at all, I'll take the criticism.

2. Voodoo

People might raise an eyebrow reading about system administration and witchery in the same context but I think most of the people who deal with computers and software know that this stuff isn't really deterministic.
A good example is when a piece of code gives you a segmentation fault and after you put debug printouts in every possible line the segmentation fault just ceases to exist. Of course, after removing the printouts everything is fine, no segmentation faults.
An even better example will be what I refer to as the "Sysadmin Effect". Systems shouldn't be treated as anything less than an intelligent being with it's own will. It might be a bit difficult to take me seriously right now, but systems really need attention like little spoiled kids. They behave as long as the sysadmin is present but start revolting as soon as they're left alone.
The system might function perfectly for weeks and months but the second you leave the office for a two day trip a processor will get burned and the phenomenon will be so irrelevant that you won't have any chance of understanding the problem on the phone (and I'm talking from experience here).
Another one. A few weeks ago I went out in the middle of the day, when I returned and inspected the monitoring software I've discovered the system had a blocking sessions issue from the minute I went out... and up until a minute ago.
Those are just two examples, I have a lot more.
You might try to convince me that there's an explanation for everything and that there's no such thing as an intelligent system - and you'll probably be right.
But me, I just call it Voodoo.

Monday, December 17, 2007

All About Attitude

In my previous post I've complained about being asked to perform major installations in order to solve small bugs.
So as I've checked the updates to my SR's coming to work after the weekend I had a classic example waiting for me. It's amazing how similar was the response to one of the SR's to mine made up example. Just proves (at least to myself, you haven't really seen the SR) I wasn't exaggerating:
"I've discovered a similar bug" (didn't look all that similar to me, though) "it has been solved in ATG_PF.H RUP5 but you might as well install RUP6".

BUT,

And you've probably noticed that's one hell of a BUT, the guy added "let me know if it's feasible".
Now, reading this have probably changed my reaction totally. Instead of getting furious at the support technician I just thought to myself: "The guy presented me with the option but he knows it's meaning and understands the implications of following this path".

Well, here is a lesson for everyone who provides support (and a sysadmin's job has definitely an aspect of support): Show the customer you care and that you understand his troubles.
It costs you nothing and you might actually get the other guy feeling a bit less frustrated.



Friday, December 14, 2007

Support, Yourself

Good support is really important. I'm a member of a sysadmins' team and I know support can really suck.
The company that supports one of the products of a fellow sysadmin uses him for support. I mean seriously, how lame can support be if they send people down to us to understand how the system they support works?
Another company (and I'm talking about a major company here) seems to ignore a serious performance issue, the problem is present on their site too but they don't seem to be doing much towards supplying a solution.

Given those, I'm used to get lots of "You got a reply to your SR within 15 mins?! I wish I had at least someone to talk to about my issue...". And indeed, the Metalink and the support system of Oracle are most impressive at times.

But people are not really satisfied with what they have, are they? I'm no different.
I have the following rule of thumb:
When I open a new SR the outcome will be one of two:

1. It's a well known issue that I somehow managed to miss searching the Metalink, there's an easy solution, we all live happily ever after.

2. It's not a well known issue.
I will be asked to perform tests that are often irrelevant, this I can live with, after all who am I to really know the insides and decide what's relevant?
But the thing that really drives me mad is when I get something like the following
"Well, the issue is not reproducible on our test environment but I see you don't have any 11.5.9 CU installed and you're only on ATG_PF.H (no RUPs) so I'm gonna ask you to install to 11.5.9 CU2 and ATG_PF.H RUP6 to match our configuration. Oh, and in two days I will write that I haven't heard back from you and so I'll inactivate this SR in one day stating I believe the issue has been resolved by the trivial solution I provided. No, I don't really think your problem will be solved but you can't prove me wrong until you try and then you'll have your hands full dealing with the mess the recently released RUP6 has caused." Does the guy really know what he's talking about? Does he really think I'll perform a semi version upgrade to maybe solve a small issue?
So, in most cases, I might get some hints from support but I will eventually have to work out the solution on my own (or with help, but not from Oracle). I might even find a piece of bugged Oracle code (I'm on Windows so wrong direction slashes might cause problems, etc.) but then I'll get something like "You're right, our bad, here's an idea for a piece of code you can write to workaround the problem".

Another annoying thing is that in both cases I'll probably be asked some questions I've already answered. What's the point of asking me all those question when I open an SR if I'm going to be asked for my R12 Rapidwiz version again by the support technician? OK, I got another technician to handle my case, can't he read the SR and see what tests I've already performed?


Actually, this post isn't meant to be (maybe just a bit) about blowing out steam about how bad is Oracle's support, it's probably relatively good.
My point is that when it comes to the real stuff it's the sysadmin's job to solve the problem, he knows his system best and that's what he's paid for. Have all the support you want but some issues will only be solved after spending many days hacking the system and getting frustrated for not achieving anything and wasting your time on annoying problems. And all by yourself.


Finally, for all the support guys who come to read this post. Nothing personal, really. I'm sure you have complaints about us customers as well, so just take it easy :-).

Friday, December 7, 2007

Weber's Law

I've started studying for my Masters in Computer Science this year and one of the courses I'm taking is Image Processing.
One of the early lectures was on color perception and so I was introduced to Weber's Law. In short, the law argues that perception of change depends on the relative change, for example, a candle lit in a dark room will probably be blinding, the same candle lit outside on a sunny day won't be noticed at all. Anyway, my explanation is probably a bit lame so just check out the link.
I always like to study about these things that show us how sophisticated human body (and nature in general) is, and how decades of research are spent to try and solve an issue that is already solved(we just have no idea how) in natural systems .

Few weeks later, I'm after a Judo session and my instructor tries to motivate one of the kids, he has to main points:
1. You're not supposed to suffer from practicing Judo, but on the other hand, you have to know what you want to achieve and realize you will have to put in some hard work to get there. Well, this is mostly a good approach to life but not what this post is about.
2. As you get better (the kid is no rookie) it's harder to tell you get better and this can sometimes be frustrating. Out of the scope and probably not subject to quantization but that's definitely a reincarnation of Weber's Law.

Actually, as I was listening to the conversation I thought it might as well be a discussion on relationships.
Only this week I realized the same applies for becoming a professional too. I recall that a month or so after I started doing my job on my own (not being a trainee anymore) I was excited to see how much I've learned and done in this relatively short period (and all - almost - on my own). Well, I'm definitely not there yet, I still learn new things daily and get thrilled by solving problems, but the sensation of progress is obviously not the same as at the beginning.
I think this is an interesting dilemma, anyone wants to be the top expert in his profession but once you get there, where's the excitement of learning new things and solving new issues?
Luckily, there are so many aspects in this line of work that one can always find something new and stimulating to learn, and this is what one should do if challenge and satisfaction is what he's after.

Personally, I think it's nearly impossible to become an expert of this caliber, but this Weber's Law stuff just got me thinking.

Thursday, November 22, 2007

Default Configuration

I always thought that default configuration should be the simplest and the easiest for implementation, well until I met Oracle default configurations.

A good demonstration of this will be describing two projects I worked on in the past year.

Some background before I begin. We've decided in our organization it's time to implement SSO for the EBS. Now, we already had the OID infrastructure required with an operable SSO configuration. It was, however, decided I should migrate the single OID server to two Load-Balanced OID servers, for High Availability purposes of course. So, first phase I migrate to Load Balanced OID servers, second phase I make the necessary EBS adjustments.
I was amazed in both phases with how much sorting out I needed to do to understand what I really need to implement.

The only documentation to work with to migrate to a Load Balanced OID layer is the documentation which shows how to migrate to a totally High-Availability configuration in each layer, so after printing out all the documentation, I had to get rid of all the RAC and such related parts and even after that there were many parts I didn't really need for the final migration procedure.

The documentation about migrating to SSO for the EBS is even more ambitious, for some reason it's assumed that the second I implement SSO I'll implement full synchronization between the EBS and the OID (probably involving the master source of identities as well). What's wrong with just using the Kerberos Ticket users get from Windows for EBS authentication? At least for beginners? In a TAR I've opened to solve some related issue I had real hard time explaining that I have no reason for registering OID since I don't want any synchronization. Actually, since it appears as such a trivial step in the installation process I had my own share of doubts on this point.

My real question is why every paper assumes I'm on RAC (although it WILL happen one of these days, probably)? What's wrong with describing basic configurations and supplying pointers to the more advanced options?

Friday, November 16, 2007

Here I am!

Well, after a period of writing a blog inside my organization I've decided to broaden my audience.

A bit about myself. I've been an Oracle Application DBA for about a year and a half now, I also work with other (mostly Oracle) applications.

A bit about this blog. In my organizational blog I'm pretty limited in getting down to the real technical stuff I sometimes want to write about, from the simple reason that almost nobody will understand what I'm writing about.
It's actually pretty exciting, in my organizational blog I may pretend writing for some general population, but all in all I know 90 percent of my readers and work with them daily. With this blog on the contrary I don't have any idea who my readers will be and if I gonna have any at all. Well, gotta take the chance!

Well, so far the first post. As you've probably already understood the next posts gonna be more technical and given my expertise, probably will deal with Oracle (Applications) related stuff.