About two weeks ago I've started training a new Apps DBA (my replacement actually). However exciting, this mission (as I was warned) is very time consuming, it's not only the time spent on explaining what does a 'system' mean and the more specific niches of the system he will have to manage, but I also, as the new guy makes progress, have to show him how to deal with day to day issues, how to apply a document to the system and how to deal with annoying developers.
Again, as in the first part of Looking At The Mirror, I find myself recalling my own training a couple of years ago and, as reminded by a friend, I remember being pretty sure there's no way I'll be able to really do something productive like my predecessor did, I thought I might be able to bring the system up at most. Well, as it is with those things, I was able to do a bit more than that. One day you just realize you're able to handle the system on your own and even more than just 'handle'.
By the way, talking about bringing up systems, today the new guy brought up two development environments on his own, I had to explain some things I've skipped before as he went along but eventually the environments were alive and kicking.
And that's another thing worth mentioning. When you know what you know, everything looks trivial. It's only when you have to transfer your knowledge to someone else that you realize how much experience you've accumulated and how tricky this all system thing is.
The bottom line is that there's really no way of transferring all you know, you can pass part of this in documents you've written, you can give oral explanations and tips, but eventually experience is the best teacher and being thrown into the deep cold water of a system crash(which will probably happen the next time I'm on vocation( - Voodoo part)) is sometimes better than weeks of explanations.
I'm pretty curious to see how the new guy handles the real stuff. For instance, if everything goes on as planned, this week he will watch while I upgrade an EBS database to 10g and in the next upgrading phase I expect him to do it on his own (well, maybe with SOME help).
Things are definitely going to get interesting... for both of us.
Sunday, January 27, 2008
Friday, January 18, 2008
Looking At The Mirror (1)
This week I've been interviewing candidates for a programmer's position.
Up until now I've always sat on the other side of the table, so it felt strange to be the one asking the questions and not the one trying to impress by answering them.
This is not a simple task.
When you grade a candidate at the end of the interview you(or at least I) not only consider his ability to answer your questions in a satisfactory way, but, inevitably, you also think whether you would grade yourself high given the considerations you've applied, I think this shouldn't be a factor and neither the fact that you tend to sympathize more with people who have a similar background to your own background. Well, we're all just human.
Actually the considerations I've applied in many of the interviews are even more "unprofessional":
The interview consists of two parts, in the first, an HR representative asks personality questions and in the second the professional interviewer(me) asks professional questions. But actually , by the time we get to the professional part you've already formed quite a firm opinion of the candidate since virtues like the ability to operate as part of a team or the ability to quickly grasp new technologies are all part of being a good programmer(and most other technological positions as well). So sometimes the professional questions are asked only in order to validate what you've already realized about a candidate.
You can probably say it's not fair to decide on a person's future (a bit dramatic but true sometimes nonetheless) based on a first impression, that's true, but life's tough and in reality our interactions with other people are often based merely on first impressions. Besides, the interview has only limited time so first impression is mostly what you have.
Up until now I've always sat on the other side of the table, so it felt strange to be the one asking the questions and not the one trying to impress by answering them.
This is not a simple task.
When you grade a candidate at the end of the interview you(or at least I) not only consider his ability to answer your questions in a satisfactory way, but, inevitably, you also think whether you would grade yourself high given the considerations you've applied, I think this shouldn't be a factor and neither the fact that you tend to sympathize more with people who have a similar background to your own background. Well, we're all just human.
Actually the considerations I've applied in many of the interviews are even more "unprofessional":
The interview consists of two parts, in the first, an HR representative asks personality questions and in the second the professional interviewer(me) asks professional questions. But actually , by the time we get to the professional part you've already formed quite a firm opinion of the candidate since virtues like the ability to operate as part of a team or the ability to quickly grasp new technologies are all part of being a good programmer(and most other technological positions as well). So sometimes the professional questions are asked only in order to validate what you've already realized about a candidate.
You can probably say it's not fair to decide on a person's future (a bit dramatic but true sometimes nonetheless) based on a first impression, that's true, but life's tough and in reality our interactions with other people are often based merely on first impressions. Besides, the interview has only limited time so first impression is mostly what you have.
Thursday, January 10, 2008
Time To Let Go
In job interviews you sometimes get asked about your bad qualities, among other things I usually mention me being a "too hard trier". It might seem like me trying to be a smart ass and slip a good quality as a bad one, but I really do think I sometimes try too hard.
I bet everyone knows this thing when you sit for hours trying to solve a problem just to get frustrated, go home and wake up with the now trivial solution the next morning. And that's exactly what I'm talking about, of course it's great to come up with a solution just when you're about to give up, but isn't it even better to skip the frustration part?
I don't really know the exact neurological explanation but a good night's sleep or even setting your mind to something else definitely triggers some processes in this head of ours that help us see the problem in a different light, and that's always a good thing. It's like when you try to remember the name of a song/movie/book and can't pinpoint it no matter how hard you try, but it will always pop into your mind later without the slightest effort.
This approach actually goes along well with what I've written in a previous post about counting on luck but somehow when it comes to letting go and letting nature do it's magic I seem to stay stubborn.
I'll try to continue my self-education and my advice to everyone who has a similar issue is to do the same, sometimes it just doesn't worth the trouble (but sometimes it does...).
I bet everyone knows this thing when you sit for hours trying to solve a problem just to get frustrated, go home and wake up with the now trivial solution the next morning. And that's exactly what I'm talking about, of course it's great to come up with a solution just when you're about to give up, but isn't it even better to skip the frustration part?
I don't really know the exact neurological explanation but a good night's sleep or even setting your mind to something else definitely triggers some processes in this head of ours that help us see the problem in a different light, and that's always a good thing. It's like when you try to remember the name of a song/movie/book and can't pinpoint it no matter how hard you try, but it will always pop into your mind later without the slightest effort.
This approach actually goes along well with what I've written in a previous post about counting on luck but somehow when it comes to letting go and letting nature do it's magic I seem to stay stubborn.
I'll try to continue my self-education and my advice to everyone who has a similar issue is to do the same, sometimes it just doesn't worth the trouble (but sometimes it does...).
Friday, January 4, 2008
Just Tell Me What Did I Do Wrong
Know that thing when your girlfriend(or wife, or whatever) starts acting like a walking icicle because you've done something to upset her? So you also know that she will put you to the test of discovering your error yourself. No matter what exactly you did you will get the same chill tone. Well, who understands women anyway?
But I've expected code to be more friendly (it's mostly written by men after all).
OK, I get it, you supply me with a wonderfully written piece of code and if I'm unable to figure out the correct parameters or set the environment correctly then I'm the stupid one here and that's my problem to deal with. But really, isn't that what Error Handling is for?
Had a variety of error handling related issues lately, here are just some of them:
1. A few weeks ago I've tried to install an R12 Vision database. After copying the installation CD (the very first phase) the installer window have just vanished. The log did contain an error code but with no explanation whatsoever and I haven't found anything on Metalink, so I tried again and this time I saw that just before crashing the installer executes bootstrap.cmd. So I tried to run it myself and got a java error stack with an error message saying something about "Unable to open a window" (I don't really remember the exact formulation right now), with no other apparent choice I've started decompiling the java code together with my team leader (who used to be an Apps DBA as well). At last we've discovered a code section where an error wasn't handled, pleased with ourselves we've logged an SR.
An unhandled error is bad enough but apparently the real problem was that the Staging Area wasn't set correctly, that's definitely my bad but how complicated it is to check the validity of the Staging Area before launching the installer and throwing totally unrelated errors at me?
2. I've noticed that it doesn't matter what's wrong, if running adcfgclone on the database tier it will probably fail with the same error stating something about "Error returned while creating control files" and an RW-50004 error code. If you're lucky the log file will have some clues, if not you'll have to start debugging Oracle's code. I got this same error for problems with an Oracle script which is part of the whole process, for problems with starting the database listener and even for a corrupted database, non of which are really related specifically to the creation of control files. And I'm pretty sure this error shows itself in much more cases. I think it's totally reasonable to expect an error message when the problem appears, instead, all errors seem to propagate to a single line of code where the adcfgclone just won't take it anymore.
3. Had to modify the virtual IP(just the IP, host name stays the same) of my load balanced OID server - turns out that's not as simple as one would expect. From what I know up until now (haven't really resolved this issue yet) it seems that the IP is hardcoded in some configuration file or maybe even in the Meta-Repository database, that's odd. Maybe I'll write about it in another post when I'll have more idea. Anyway, in the process I've discovered a script that's supposed to do just that, it even has it's own folder directly under ORACLE_HOME. This fact made me suspicious at once since it makes this script look like a late add-on which wasn't considered with much thought. The code inside really stands up to the expectations, it seems no one have ever tested the code. It looks something like this:
...
if
echo bad thing 1 happened
goto end
...
if
echo bad thing 2 happened
goto end
...
...
:end
exit
Doesn't take a cmd expert to understand that running the script as it is won't show you any error messages (unless you're a really quick reader) and it will actually close your cmd window (why?!).
Well, I think that concludes my case.
But I've expected code to be more friendly (it's mostly written by men after all).
OK, I get it, you supply me with a wonderfully written piece of code and if I'm unable to figure out the correct parameters or set the environment correctly then I'm the stupid one here and that's my problem to deal with. But really, isn't that what Error Handling is for?
Had a variety of error handling related issues lately, here are just some of them:
1. A few weeks ago I've tried to install an R12 Vision database. After copying the installation CD (the very first phase) the installer window have just vanished. The log did contain an error code but with no explanation whatsoever and I haven't found anything on Metalink, so I tried again and this time I saw that just before crashing the installer executes bootstrap.cmd. So I tried to run it myself and got a java error stack with an error message saying something about "Unable to open a window" (I don't really remember the exact formulation right now), with no other apparent choice I've started decompiling the java code together with my team leader (who used to be an Apps DBA as well). At last we've discovered a code section where an error wasn't handled, pleased with ourselves we've logged an SR.
An unhandled error is bad enough but apparently the real problem was that the Staging Area wasn't set correctly, that's definitely my bad but how complicated it is to check the validity of the Staging Area before launching the installer and throwing totally unrelated errors at me?
2. I've noticed that it doesn't matter what's wrong, if running adcfgclone on the database tier it will probably fail with the same error stating something about "Error returned while creating control files" and an RW-50004 error code. If you're lucky the log file will have some clues, if not you'll have to start debugging Oracle's code. I got this same error for problems with an Oracle script which is part of the whole process, for problems with starting the database listener and even for a corrupted database, non of which are really related specifically to the creation of control files. And I'm pretty sure this error shows itself in much more cases. I think it's totally reasonable to expect an error message when the problem appears, instead, all errors seem to propagate to a single line of code where the adcfgclone just won't take it anymore.
3. Had to modify the virtual IP(just the IP, host name stays the same) of my load balanced OID server - turns out that's not as simple as one would expect. From what I know up until now (haven't really resolved this issue yet) it seems that the IP is hardcoded in some configuration file or maybe even in the Meta-Repository database, that's odd. Maybe I'll write about it in another post when I'll have more idea. Anyway, in the process I've discovered a script that's supposed to do just that, it even has it's own folder directly under ORACLE_HOME. This fact made me suspicious at once since it makes this script look like a late add-on which wasn't considered with much thought. The code inside really stands up to the expectations, it seems no one have ever tested the code. It looks something like this:
...
if
echo bad thing 1 happened
goto end
...
if
echo bad thing 2 happened
goto end
...
...
:end
exit
Doesn't take a cmd expert to understand that running the script as it is won't show you any error messages (unless you're a really quick reader) and it will actually close your cmd window (why?!).
Well, I think that concludes my case.
Friday, December 28, 2007
Gotta Have Faith
I'm an atheist.
But I really do understand the need to have something powerful and almighty to believe in. Especially since I'm a sysadmin. How come? Well, the way I see it, two of the reasons people invented god are:
1. People need something to depend on in hard times.
2. People need explanations for the unnatural.
Both apply to sysadmins.
I'll elaborate.
1. Counting on Luck
Sometimes life gets tough. Sometimes you have an issue with some installation that you can't solve (and from how things evolve you know the help won't come from support), you're way beyond schedule and more problems keep rushing at you.
In times like this I tend to tell myself, like a good friend of mine phrased a while ago, "In the very very end things will be OK". Actually, now that I read it, it sounds a bit pessimistic but the intention is actually the opposite, the "very very end" doesn't have to sound so fatal, it just means problems will be solved in their own time. Sometimes, one just have to realize that no matter how grim a situation is, the issue WILL be solved, there's really no way you're not going to succeed in upgrading the god-forsaken database! So why worry (too much)?
This approach might make me sound a bit of an indifferent person, but if that what keeps me going after spending a couple of weeks trying to solve an issue and not making any progress at all, I'll take the criticism.
2. Voodoo
People might raise an eyebrow reading about system administration and witchery in the same context but I think most of the people who deal with computers and software know that this stuff isn't really deterministic.
A good example is when a piece of code gives you a segmentation fault and after you put debug printouts in every possible line the segmentation fault just ceases to exist. Of course, after removing the printouts everything is fine, no segmentation faults.
An even better example will be what I refer to as the "Sysadmin Effect". Systems shouldn't be treated as anything less than an intelligent being with it's own will. It might be a bit difficult to take me seriously right now, but systems really need attention like little spoiled kids. They behave as long as the sysadmin is present but start revolting as soon as they're left alone.
The system might function perfectly for weeks and months but the second you leave the office for a two day trip a processor will get burned and the phenomenon will be so irrelevant that you won't have any chance of understanding the problem on the phone (and I'm talking from experience here).
Another one. A few weeks ago I went out in the middle of the day, when I returned and inspected the monitoring software I've discovered the system had a blocking sessions issue from the minute I went out... and up until a minute ago.
Those are just two examples, I have a lot more.
You might try to convince me that there's an explanation for everything and that there's no such thing as an intelligent system - and you'll probably be right.
But me, I just call it Voodoo.
But I really do understand the need to have something powerful and almighty to believe in. Especially since I'm a sysadmin. How come? Well, the way I see it, two of the reasons people invented god are:
1. People need something to depend on in hard times.
2. People need explanations for the unnatural.
Both apply to sysadmins.
I'll elaborate.
1. Counting on Luck
Sometimes life gets tough. Sometimes you have an issue with some installation that you can't solve (and from how things evolve you know the help won't come from support), you're way beyond schedule and more problems keep rushing at you.
In times like this I tend to tell myself, like a good friend of mine phrased a while ago, "In the very very end things will be OK". Actually, now that I read it, it sounds a bit pessimistic but the intention is actually the opposite, the "very very end" doesn't have to sound so fatal, it just means problems will be solved in their own time. Sometimes, one just have to realize that no matter how grim a situation is, the issue WILL be solved, there's really no way you're not going to succeed in upgrading the god-forsaken database! So why worry (too much)?
This approach might make me sound a bit of an indifferent person, but if that what keeps me going after spending a couple of weeks trying to solve an issue and not making any progress at all, I'll take the criticism.
2. Voodoo
People might raise an eyebrow reading about system administration and witchery in the same context but I think most of the people who deal with computers and software know that this stuff isn't really deterministic.
A good example is when a piece of code gives you a segmentation fault and after you put debug printouts in every possible line the segmentation fault just ceases to exist. Of course, after removing the printouts everything is fine, no segmentation faults.
An even better example will be what I refer to as the "Sysadmin Effect". Systems shouldn't be treated as anything less than an intelligent being with it's own will. It might be a bit difficult to take me seriously right now, but systems really need attention like little spoiled kids. They behave as long as the sysadmin is present but start revolting as soon as they're left alone.
The system might function perfectly for weeks and months but the second you leave the office for a two day trip a processor will get burned and the phenomenon will be so irrelevant that you won't have any chance of understanding the problem on the phone (and I'm talking from experience here).
Another one. A few weeks ago I went out in the middle of the day, when I returned and inspected the monitoring software I've discovered the system had a blocking sessions issue from the minute I went out... and up until a minute ago.
Those are just two examples, I have a lot more.
You might try to convince me that there's an explanation for everything and that there's no such thing as an intelligent system - and you'll probably be right.
But me, I just call it Voodoo.
Monday, December 17, 2007
All About Attitude
In my previous post I've complained about being asked to perform major installations in order to solve small bugs.
So as I've checked the updates to my SR's coming to work after the weekend I had a classic example waiting for me. It's amazing how similar was the response to one of the SR's to mine made up example. Just proves (at least to myself, you haven't really seen the SR) I wasn't exaggerating:
"I've discovered a similar bug" (didn't look all that similar to me, though) "it has been solved in ATG_PF.H RUP5 but you might as well install RUP6".
BUT,
And you've probably noticed that's one hell of a BUT, the guy added "let me know if it's feasible".
Now, reading this have probably changed my reaction totally. Instead of getting furious at the support technician I just thought to myself: "The guy presented me with the option but he knows it's meaning and understands the implications of following this path".
Well, here is a lesson for everyone who provides support (and a sysadmin's job has definitely an aspect of support): Show the customer you care and that you understand his troubles.
It costs you nothing and you might actually get the other guy feeling a bit less frustrated.
So as I've checked the updates to my SR's coming to work after the weekend I had a classic example waiting for me. It's amazing how similar was the response to one of the SR's to mine made up example. Just proves (at least to myself, you haven't really seen the SR) I wasn't exaggerating:
"I've discovered a similar bug" (didn't look all that similar to me, though) "it has been solved in ATG_PF.H RUP5 but you might as well install RUP6".
BUT,
And you've probably noticed that's one hell of a BUT, the guy added "let me know if it's feasible".
Now, reading this have probably changed my reaction totally. Instead of getting furious at the support technician I just thought to myself: "The guy presented me with the option but he knows it's meaning and understands the implications of following this path".
Well, here is a lesson for everyone who provides support (and a sysadmin's job has definitely an aspect of support): Show the customer you care and that you understand his troubles.
It costs you nothing and you might actually get the other guy feeling a bit less frustrated.
Friday, December 14, 2007
Support, Yourself
Good support is really important. I'm a member of a sysadmins' team and I know support can really suck.
The company that supports one of the products of a fellow sysadmin uses him for support. I mean seriously, how lame can support be if they send people down to us to understand how the system they support works?
Another company (and I'm talking about a major company here) seems to ignore a serious performance issue, the problem is present on their site too but they don't seem to be doing much towards supplying a solution.
Given those, I'm used to get lots of "You got a reply to your SR within 15 mins?! I wish I had at least someone to talk to about my issue...". And indeed, the Metalink and the support system of Oracle are most impressive at times.
But people are not really satisfied with what they have, are they? I'm no different.
I have the following rule of thumb:
When I open a new SR the outcome will be one of two:
1. It's a well known issue that I somehow managed to miss searching the Metalink, there's an easy solution, we all live happily ever after.
2. It's not a well known issue.
I will be asked to perform tests that are often irrelevant, this I can live with, after all who am I to really know the insides and decide what's relevant?
But the thing that really drives me mad is when I get something like the following
"Well, the issue is not reproducible on our test environment but I see you don't have any 11.5.9 CU installed and you're only on ATG_PF.H (no RUPs) so I'm gonna ask you to install to 11.5.9 CU2 and ATG_PF.H RUP6 to match our configuration. Oh, and in two days I will write that I haven't heard back from you and so I'll inactivate this SR in one day stating I believe the issue has been resolved by the trivial solution I provided. No, I don't really think your problem will be solved but you can't prove me wrong until you try and then you'll have your hands full dealing with the mess the recently released RUP6 has caused." Does the guy really know what he's talking about? Does he really think I'll perform a semi version upgrade to maybe solve a small issue?
So, in most cases, I might get some hints from support but I will eventually have to work out the solution on my own (or with help, but not from Oracle). I might even find a piece of bugged Oracle code (I'm on Windows so wrong direction slashes might cause problems, etc.) but then I'll get something like "You're right, our bad, here's an idea for a piece of code you can write to workaround the problem".
Another annoying thing is that in both cases I'll probably be asked some questions I've already answered. What's the point of asking me all those question when I open an SR if I'm going to be asked for my R12 Rapidwiz version again by the support technician? OK, I got another technician to handle my case, can't he read the SR and see what tests I've already performed?
Actually, this post isn't meant to be (maybe just a bit) about blowing out steam about how bad is Oracle's support, it's probably relatively good.
My point is that when it comes to the real stuff it's the sysadmin's job to solve the problem, he knows his system best and that's what he's paid for. Have all the support you want but some issues will only be solved after spending many days hacking the system and getting frustrated for not achieving anything and wasting your time on annoying problems. And all by yourself.
Finally, for all the support guys who come to read this post. Nothing personal, really. I'm sure you have complaints about us customers as well, so just take it easy :-).
The company that supports one of the products of a fellow sysadmin uses him for support. I mean seriously, how lame can support be if they send people down to us to understand how the system they support works?
Another company (and I'm talking about a major company here) seems to ignore a serious performance issue, the problem is present on their site too but they don't seem to be doing much towards supplying a solution.
Given those, I'm used to get lots of "You got a reply to your SR within 15 mins?! I wish I had at least someone to talk to about my issue...". And indeed, the Metalink and the support system of Oracle are most impressive at times.
But people are not really satisfied with what they have, are they? I'm no different.
I have the following rule of thumb:
When I open a new SR the outcome will be one of two:
1. It's a well known issue that I somehow managed to miss searching the Metalink, there's an easy solution, we all live happily ever after.
2. It's not a well known issue.
I will be asked to perform tests that are often irrelevant, this I can live with, after all who am I to really know the insides and decide what's relevant?
But the thing that really drives me mad is when I get something like the following
"Well, the issue is not reproducible on our test environment but I see you don't have any 11.5.9 CU installed and you're only on ATG_PF.H (no RUPs) so I'm gonna ask you to install to 11.5.9 CU2 and ATG_PF.H RUP6 to match our configuration. Oh, and in two days I will write that I haven't heard back from you and so I'll inactivate this SR in one day stating I believe the issue has been resolved by the trivial solution I provided. No, I don't really think your problem will be solved but you can't prove me wrong until you try and then you'll have your hands full dealing with the mess the recently released RUP6 has caused." Does the guy really know what he's talking about? Does he really think I'll perform a semi version upgrade to maybe solve a small issue?
So, in most cases, I might get some hints from support but I will eventually have to work out the solution on my own (or with help, but not from Oracle). I might even find a piece of bugged Oracle code (I'm on Windows so wrong direction slashes might cause problems, etc.) but then I'll get something like "You're right, our bad, here's an idea for a piece of code you can write to workaround the problem".
Another annoying thing is that in both cases I'll probably be asked some questions I've already answered. What's the point of asking me all those question when I open an SR if I'm going to be asked for my R12 Rapidwiz version again by the support technician? OK, I got another technician to handle my case, can't he read the SR and see what tests I've already performed?
Actually, this post isn't meant to be (maybe just a bit) about blowing out steam about how bad is Oracle's support, it's probably relatively good.
My point is that when it comes to the real stuff it's the sysadmin's job to solve the problem, he knows his system best and that's what he's paid for. Have all the support you want but some issues will only be solved after spending many days hacking the system and getting frustrated for not achieving anything and wasting your time on annoying problems. And all by yourself.
Finally, for all the support guys who come to read this post. Nothing personal, really. I'm sure you have complaints about us customers as well, so just take it easy :-).
Subscribe to:
Posts (Atom)