Just Stupid

Well, I was actually going to write the second (and probably final) part of Light at the End of the Tunnel but some things are just too provoking (and I'm not at the end of the tunnel yet anyway).

My folks bought a new phone device today, probably the cheapest device in the whole store, but that's not really a justification for the piece of crap this device turned out to be.
What can be so terrible about a simple phone device? Well, I'll supply an example, and please keep in mind that this is after only a ten minutes interaction with the device, I'm actually a bit curious to discover what other flaws the device has...
To the example then. My dad opens the box, connects the phone and tries to dial... nothing. Apparently someone had a "brilliant" idea, the device has a lock (see picture below) that somehow reminds the ages when computers used to had locks - didn't find an image of that one (and didn't look to hard either), anyway, the purpose of the lock is to block calls to certain numbers: it has three modes - dial any number, don't dial any number and dial only numbers that don't start with an '0'. Those three modes are represented by three colors: red, yellow and green, at least it's trivial which color represents which mode, that's right, red is "dial anywhere" and green is "forget about dialing", someone else thinks that's extremely stupid? If you take a close look at the picture you'll also notice that there's no real way to know at what position the lock is now (yeah, it's not green), very amusing. What adds to the amusement, considering the fact that the phone didn't came "red" by default, is that the Hebrew version of the user's guide doesn't have a single word about the lock.
And if that wasn't enough I have another example. The device has a speaker, so you'd probably expect to be able to dial a number, press the speaker button and have a conversation rolling, not with this one. Once you press the speaker button the number typed disappears and you're welcome to type in a new number, so you have to press the speaker button before dialing or use some commando techniques such as pressing the redial button after the number you typed has dissolved - that's a bit weird since you didn't really "dial" that number before, oh well.

A home phone device is supposed to be as simple as possible to use and trivial to understand for everyone, this one has definitely failed this test. Just Stupid.

Alerts

Last weekend I woke in the middle of the night to discover my arm totally numb, and when I say "totally" I really mean it - I couldn't feel it nor move it.
I don't know if I'm some kind of freak or maybe it happens to everyone occasionally but it's definitely not the first time it has happened to me. What I usually do when it happens is to grab my "dead" arm with my other arm and throw it off the side of the bed - I do it to hasten the blood flow back into my arm (have no idea if this actually has any effect though) - this is actually the creepiest part of this whole process, because when I grab my numb arm with my not-numb arm I don't feel like touching myself and the arm is simply dangling like a dead piece of meat until blood flow is restored.
Up until now the arm revived each time, but I always wonder to myself what would've happened have I not woken up. I also wonder if what got me awake is some alerting (hence the title) mechanism of my body, and since I don't tend to wake up in the middle of the night this is probably the case.

What this short tale got to do with system administration (except happening to a sysadmin), well, nothing really. But it did got me thinking that I should write a post about the different monitoring and administration tools I use to detect and analyze bad things in my system, of course, I still care more about my own arm - can't type commands too fast with just one arm :).

Information Boom

I'll take another break from my upgrade aftermath tales to write about an issue I think many sysadmins face, and not only sysadmins. Handling information.

It seems that since I started this job I'm in a constant race to beat the accumulation of information, and I never seem to be on top.
When I started my training I had a notebook in which I wrote all my remarks, since at the beginning you write any minor note (although later it's trivial anyway) I soon started a second notebook. But notebooks aren't so efficient so after starting the job for real I've started to use OneNote, WOW! I can create plenty of pages in a single tab, this will definitely do the trick! But soon I needed more tabs and then more OneNote notebooks too.
To store documents my team (a team of sysadmins) uses a SharePoint portal, we used to have a folder-hierarchy structure to our documents, but at some point it just got too messy so we've decided just to throw everything in the root folder (chaos indeed) and tag it, then of course we've created views and filters for the different tags.
We also have a Wiki to store and share important knowledge.

But even with all those mechanisms set in place it seems there's just too much information to handle:
I have my daily tasks to manage. Every new topic (regardless how minor) I deal with is a page or a tab in my OneNote. A major upgrade or installation consist of my own notes (what I did, what errors I got), official installation/upgrade notes, know bugs documents, new features' documents etc.. Every problem has a related metalink note or my own document describing how to fix it. And except the EBS I'm also in charge of an OID and an iAS servers.
That's just a partial list and nothing in it can be skipped.
Both external(e.g. supplied by the vendor) and internal (written by the sysadmin) documentation is critical to every sysadmin - you can't memorize every parameter and functionality in your system and you can't expect to remember every error you handled three years ago.
But the problem starts when you need to retrieve this information, if you don't organize your information wisely it doesn't matter that you have a perfect documentation of how to fix a bug because you just won't be able to find it, too bad. I can think of many times when I was sure I saw something familiar before and had to start guessing where exactly - my OneNote? The portal? Maybe in my documents' folder before I started using the portal? Maybe I eventually didn't download it from the metalink? Or maybe someone else showed it to me? Trust me, this can get pretty frustrating.
Having a good logic for storing your information is crucial but from my experience it's just not enough as information tends to outgrow any such solution.
As I see it, the only real solution is a powerful searching solution that has access to all the storage solutions you use.

I'm actually going to explore this field of enterprise search solutions so I might have more insight in following posts.

Light at the End of the Tunnel, Part I

This post should actually be titled "Upgrading Part IV", but my Roman math is not so good - I admit checking if I should write IV or VI - so I decided to stick to the simple numbers.

After bitching about it a lot, it's time I tell you how my inconsistent problem got solved.
This one was too much for me to solve on my own so I had an Oracle expert come over. After describing the issue I'm facing he said we should set up a trace on all client connections from the 8.0.6 home - we had to do this for all connections since we had no idea which concurrent will get stuck. It took some time before we realized that we had too small a pool of trace files and that trace files we were looking at didn't belong to a stuck concurrent anymore. After doing some re-configuration on the trace settings we finally found a trace file that belonged to a stuck concurrent.
Boy! The guy has patience. As someone on my team said, a good expert can be measured by the patience he demonstrates going over log and trace files.
Going over the entire trace file we discovered that at some point during connection establishment the process just stops, we could see that the client was waiting for some input and... nothing. Although it felt like closing on on the problem it seemed like a long and tedious process will be needed through a TAR, since however good the expert I had on site is, it's just not his specialty, and it definitely looked like some internal bug.
Before leaving, the expert noticed I had NTS defined in my sqlnet.ora file for SQLNET.AUTHENTICATION_SERVICES, I explained I was using it to logon as sysdba without entering a password. He suggested I try to remove it, so I did.
At this point we were towards the end of the day so I didn't expect any concurrents to get stuck anyway. But the next day the problem disappeared, I was thrilled, seems like the NTS setting was the problem all the way! Thinking about it, there's no chance I would've thought of it to be the root of all evil. I think the only reason the expert noticed it is because he's not used to dealing with installations on Windows, it seems to me that even he didn't believe it will solve anything and suggested it just as a shot in the dark.
Proper disclosure. The expert suggested some other changes as well. Those were changes to DB parameters as specified in an Oracle document regarding DB parameters for an EBS installations.
The changes had effect only on the next day (the day everything worked) as well, so I might be wrong and what really fixed the problem was altering those parameters. Since I'm curious and somewhat suicidal I'll probably have the NTS returned and check if the problem reoccurs, promise to let know if my intuition here is wrong.

The funny thing is that just after discovering this issue with NTS, I found another one, working on an entirely different thing. Try googling for "NTS sqlnet.ora".

Well, one problem down, one more to go. Probably will tell about it in my next post, although it's still not solved.

Widening My Horizons

Well, I've decided to take a break from my series of upgrade related posts and write about something more fun.

This week I got a new 22'' (wide) monitor as a replacement for my old 17'' monitor. No doubt this is a great improvement, so many things look different now:
Most emails fit the screen without scrolling, I can accommodate more tabs in my Total Commander, I can comfortably rearrange my OneNote pages - it actually feels like each page is twice as wide, I can open two Terminal Services side by side and work with them in parallel, it's easier to read documents - less scrolling, and those are just the things I could think of on the spot after one week's work. I'm sure I'll discover many more advantages. Oh, the joy of having a new toy!

Of course, there's a downside as well(not that I really care, but...), I'm going to get spoiled.
I think it's relevant to many aspects in life, when you get a taste of a better something, you can't really go back to what you previously did just fine with - and yes, I was pretty satisfied with my 17'', although getting an occasional "man, how can you work with it?!" from guys with bigger monitors. And now I'm going to become such a guy.
It reminds of when I was a kid and played Warcraft (the 1st one of course). I played it on my computer and was just fine with it, but then I was at a friend's house and he had a faster PC, after playing a few levels on his PC mine seemed so sloooow, it was impossible to play.

My conclusion is that such improvements are good only if you know you won't have to revert back to the previous situation, otherwise it's going to take a long time getting used to and might even be frustrating.
I guess I'm going to demand a 22'' monitor to be included in every contract I sign from now on...

Upgrading Part III: Consistency

Well, I'm still battling problems after the database upgrade.
I think that the most annoying thing about the problems I'm facing is that they're not really consistent.

Of course, we'd all like a problem free world, but once we encounter a problem the first thing to do is to reproduce it, thus assuring ourselves we know what we're dealing with.
First you try to reproduce it on a development environment if that doesn't succeed, you at least expect some consistency on the production environment.
When you don't have this either you're in a trouble, you might do something one day and hope to see results on the next day and indeed the next day problem still occurs but in much lower rates and you think to yourself "I must be in the right direction" but then the next day business is as usual - the system is in a total mess. Then you understand you don't understand anything. The next step is to try and reproduce problem on your terms.
In my case the issue is with concurrents randomly getting stuck. So I first tried to see if the problem is with all executables or is EBS related, starting with simple cmd scripts running non-stop and going to submitting multiple concurrents using CONCSUB - at last one of my manually submitted concurrents hangs. But I got no real insights from that as well. No consistency whatsoever.
One of the main issues with inconsistent problems is that they're hard to describe - in a TAR for example - most directions for detecting the problem involve some major configuration problem, one that would prevent the system from working altogether, but remember, I don't have this kind of blessed consistency.
Another thing with inconsistent issues is that you come to work and hope things will get bad so you could poke the issue around a bit more and so you won't get that annoying voodoo feeling. Usually a sysadmin's hope is that nothing will get bad...

Actually, this week I had some progress, so hopefully the next post will contain a bit of advice that can potentially save a serious headache to someone if he stumbles upon this blog while trying to resolve a similar issue.

Upgrading Part II: Bad Job

As with any upgrade/major installation I've ever made (at least as far as I can remember), after upgrading the database to 10g the application have demonstrated some interesting errors. One of the problems is related to concurrents and the executables they spawn crashing in mid-air - promise to tell about this in more detail when I myself have any idea.
At some point I suspected that the problem was with custom code running cmd scripts. Since some of those scripts are called from within a PL/SQL code using a Java stored procedure I thought that maybe I should try and use a "less custom" way to do that.

Luckily (or unluckily), I just read the previous week about a new feature in 10g - the dbms_scheduler package that's supposed to replace dbms_job and to be much more powerful, for instance it enables you to run cmd scripts. So I thought I'll try this out since it sounds exactly like the built-in method I was looking for.
Well, I have only one thing I can say about this: it's better to leave a feature out of the release than to keep it in when it sucks, totally.

Really, my keyboard is still soaked with sweat from my efforts to run a single script that echoes some text.
I've already written about this but I guess the message didn't get through, if I'm supposed to start the Oracle Scheduler service to run jobs then that's exactly what I expect to be written in the error message I get when not doing so - certainly not a "file not found error". I also expect this to be written in any (well, at least some) documentation describing the new feature, that's not the kind of things one supposed to dig up only in forums. By the way, some hilarious problem with similar symptoms I read about while searching for a solution to my own issue - I've actually tried this out since at first I thought this was the problem I was experiencing - it appears that for some users (maybe in earlier 10g versions) just supplying a cmd script didn't work, they had to run cmd.exe with parameters. Can't even begin to understand how you manage to create this bug and release it.
Well, after completing the POC (if at a the cost of health) I got to the real thing running a script with parameters. In some document I saw something like the following example:

dbms_scheduler.create_job(job_name => 'JOB', job_type => 'EXECUTABLE',
job_action => 'script.cmd', number_of_arguments => n);
dbms_scheduler.set_job_argument_value(job_name => 'JOB',
argument_position => 1, argument_value => '...');
...
dbms_scheduler.set_job_argument_value(job_name => 'JOB',
argument_position => n, argument_value => '...');
dbms_scheduler.enable (name => 'JOB');
dbms_scheduler.run_job(job_name => 'JOB');

Well, maybe it's just my system that is a freak but apparently the enable procedure erases the job. Exactly, I create a job, I run dbms_scheduler.enable and no job in the table - no error message, no nothing, I might have run drop_job instead. Well, apparently I don't need that line anyway. After some more struggle with cryptic error messages I got the package to do what I wanted it to do, but that's really not good enough.
I can't even start to imagine what kind of efforts are needed to bootstrap all the advanced scheduling features - windows, chains etc..

I'm willing to bet money on the fact that most developers would have given up much earlier than I did saying this stuff just doesn't work. You can develop useful and cool new features all you like but if you can't cut/paste an example and see it just working no one would use it, if all users get when something is wrong are misleading error messages they'll just get frustrated (and you can see I am one such user) .
I first read about this feature in a "10g Top 20 New Features" document, and indeed it sounded great but if it's impossible to use it, it's not really a new feature at all.
I'm sometimes not sure if I should account all those funny errors I deal with to the fact that my system is on MS Windows - a less common platform, maybe I should. But that's not a good enough reason, if I'm in possession of a disk labeled 10.2.0.3 for Windows I expect it to work, I don't really mind having it released half a year later than the corresponding Linux version, I just want it to function.

As for me, I know about this feature, I can even make it work, but there's no chance I'll suggest it as a solution to any need unless as a last resort. Too bad.