Sysadmin Tales

Wrong Conclusion

2011-02-18T16:57:00.003+02:00

I've concluded my previous post by saying that our organization should probably use EMC for every system, I was berated for this conclusion and with a good reason - I was wrong.

Yes - if stability is the only consideration so sticking with EMC is a very good idea, but it's not.

There are also other considerations, like money - EMC (and it's maintenance) is expansive and many times you need a lot of storage for an unimportant purpose. There's also the fact that there's a limitation on how many servers you can physically connect to the EMC.

So the right answer is, like always, somewhere in between, trying to meet all the needs with one solution is very ambitious, there should be a few solutions available - not a single one and not dozens of them. The important thing is that when you arrive to a point where you decide to use a new solution:

You do it for a good reason.
You "take care of it" - you need to test it, monitor it and have a standard for maintaining it, just like you have with any other solution.

Hopefully this atones for my sin...

Multiple Points of Failure

2010-12-31T12:11:00.003+02:00

I wrote in an earlier post about how sometimes it's best to rely on well known technologies rather than on carefully tailored solutions. Last week we had just the perfect example for this concept and this time from the hardware domain.

So what happened?

One of our system crashed and won't boot, probably because of a major OS failure since the mirrored disc seemed to be corrupted as well, the only guess we have for the underlying cause is lack of space on the C: drive but we don't really have a way of knowing for sure post-mortem.

The system was all on this single disc so we had to use the backup. Bad news, apparently the last backup was from like a month ago. Recovering data from the corrupted discs we've managed to narrow the damage to two weeks of lost data, but that's not the point, losing data is something that should never happen and we really try hard to prevent those kind of things from happening.

So where did we go wrong? Well, most of our systems use EMC for storage, not in this case, why?

Usually when we configure a new system we ask ourselves two main questions:

1. Is the system gonna need an SRDF/CE configuration?

2. Is the required storage for the system is more than what comes with the target server's disk drives?

If the answer to either question is YES, we use EMC otherwise we usually use it only as a location for storing the backup of the system. In our case it was the 'otherwise' option.

The problem is that when we use EMC for storage, we also use EMC's RM to take backup and have an already existing system of emails that let us know if the backup was successful.

In our case not only the data wasn't on the EMC but on a local disk but also monitoring on backups didn't exist, of course it is our bad, but that's exactly what I was talking about when I said that it's better to rely on something you're well familiar with.

I guess one of the conclusions is to try and use EMC for every system no matter how low the high availability requirements are and how little disk space is required.

Exploring

2010-09-11T11:44:00.001+03:00

One of the most basic things one does with his OS is managing files, this task becomes critical if you have to access multiple file locations - I know it's critical for me, since I manage files on many different servers and on several local locations during a typical workday.

The basic requirements for a file managing utility are pretty much well, ehh... basic: just give me something to comfortably view and transfer files. There are of course a lot of nice additional features: packing/unpacking, good search capabilities, convenient selecting, etc., but for start just give me something I can work with.

Back to my first sentence: managing files should be basic, and well in my humble opinion, not that complicated. So how come Microsoft still have the crippled Windows Explorer?!

Really, unless I'm missing some patent the whole world has I don't understand how come Explorer doesn't implement the most basic feature even the old Norton Commander had - two panes with which you can copy files between locations or easily see the disk's information. Without this so basic ability I don't know if there's really a need to mention other drawbacks (however, I will) like:

The XP's\2003 (didn't test it on Windows 7 because I kinda lost hope) totally not working searching within documents feature. I mean, OK let the search be slow, inefficient, whatever, but how can it miss a simple phrase in a .txt file?
How do I select all *.foo files in a folder? I'd be very happy to hear a simple answer to that as I really don't know how this is done, and it seems such a basic task...
And if it's even possible Windows7\2008 interface is even less user(\me) friendly, didn't really understand what they were trying to do.
Well, you got the point by now.

As for me, I've been working with FAR Manager for many years, but today my weapon of choice is Total Commander, obviously it's not perfect but it does the job and does it good. One of many features it has (and Explorer doesn't) is the tabbed interface, I won't say it's genius because I think it's simply a trivial requirement from something that wasn't written 5 years ago.

The thing that most amazes me is that most people still use the old good(?) Explorer, some maybe I'm just hard to please...

Performance

2010-08-21T12:40:00.008+03:00

Every sysadmin has to deal with performance issues once in a while.

As the "the sysadmin of sysadmins" I'm dealing a lot with allocating resources to VMs the developers for the different systems use. This is not a simple task since I have limited physical hardware and about 100 virtual development servers to manage and of course everyone wants his VMs to be as strong as needed. Or more accurately as desired.

The thing is that from my experience, the strength you need for a development server is usually far below official specs since you don't experience the same load on development environments and you don't really need everything to be as smooth as with a production environment.

Of course we do experience more serious performance issues on development environments - especially with new systems we are not fully familiar with yet, but it seems in most cases those issues are not solved by pimping up the server.

I've found myself more than once trying to explain that there's a "real" performance issue and that someone (depending on the case) has to take a really good look at the code, the design or whatever and find the problem, because it's not going to be solved by adding cores, RAM or using a physical server. In those cases I usually have to supply a proof that the RAM and CPU are sufficient and even then I have to deal with claims like "but it's a VM, it might be lying".

It's really like trying to solve an NP-Complete problem in a brute force technique by adding more computational power - this is not a real solution if your input is not bounded and things are never really this simple...

History

2010-07-26T21:28:00.002+03:00

Only last week I've completed an upgrade I've been working on for a long long time - working on a low priority project while leading a team doesn't contribute to agile upgrading. At least I've learned some things since back when.

Interestingly enough the Apache .dll error didn't reproduce on other development environments. Up until the moment of truth - the production upgrade. Obviously I had no recollection of the solution, I did remember running into this error and had it documented in my notes but not the solution - shame on me. Luckily, this time this error was documented in Metalink and the solution was much simpler- there was a missing Windows .dll (msvcr70.dll if I'm not mistaken).

But the more interesting issue is the one that prevented OPMN startall CA from completing...

After opening an SR (and a lot of debugging) I discovered the issue was with "ldapbinding" on the SSL port taking a long long time, in other words some heavy load on this port. I was advised starting the oidldapd process with more dispatchers (I later learned adding more workers worked better for me), that did solve the issue but this is not what baffles me about this case.

The first OID server I upgraded didn't suffer from this issue, I used to attribute it to the fact that it was the only server not being part of an OID cluster - although because of the nature of the upgrade process it always seemed like a lame excuse.

The thing is... the first production server I've upgraded was OK as well. There's now only one thing I can think of common to both servers: they both weren't installed by me. My memory fails me again here, since there might be an additional server I didn't install and did had the problem, but there's a good chance it was installed in a slightly different way, a way I don't think should affect oidldapd's function but still a different way.

The thought that drives me crazy right now is what kind of different installation steps my ancestors could have taken that matter so much, all I did was following their (with time evolved) documentation. Or maybe they did do some secret configuration steps they left no trace of???

Temporary

2010-07-17T17:45:00.002+03:00

One of the things I've learned being a sysadmin is that there's no such thing as a "temporary solution", if you're presented with one or thought about one yourself first ask yourself how do you feel about this solution becoming permanent, because it's here to stay.

Reasons for temporary solutions becoming permanent are numerous, but I think most of them come down to the (reasonable) claim that it's better to turn efforts to something that needs fixing than to something that already works. Many times the burden of maintaining such a temporary solution falls on the sysadmin so one else sees how bad it really is and sometimes it's just more efficient (in the immediate time frame) to accept this burden than to solve the issue once and for all.

One example I always fall back to when I have an argument about a temporary solution is the example of the email sending mechanism we use in our Oracle Applications system:

Our basic solution for sending email is based on an EBS alert that monitors a table we insert data into when we want the system to send an email to someone. An important detail: an EBS alert is simply a database trigger that is wrapped with a test that the table was updated by APPS - the EBS administrative user.

Now, several years ago we've started using ApEx as a platform for developing applications over our EBS. The catch - ApEx applications don't use the APPS user so they can't send email the way we used to, at least not without modification to our mechanism. "Luckily", ApEx has it's own email sending mechanism so we decided to use it as a temporary solution. That was about three years ago.

At this point you probably can ask me "well, who cares?", and I'll probably answer "well, I do". I do, because when we upgraded our mail server to Exchange 2007 it didn't really like the Hebrew encoding ApEx was using and decided not to display it properly. So after a day or two in which emails just weren't sent from the affected ApEx applications we had to devise a solution I'm not sure anyone will know about when the Exchange server will be upgraded to 2010.

It's a shame because the solution to this is issue was designed a long time ago, the ApEx mechanism causes application pages to hang while mail is being sent, but still - why fix something that isn't broken?

Beware of temporary solutions.

Copy-Paste

2010-06-15T21:07:00.004+03:00

I can't think of many features that are both absolutely necessary and the biggest threat to the world as we know it (well, almost) at the same time as the ability to copy and paste text at the blink of a Ctrl+C\V.

Retyping (or rewriting) a text over and over again is something computers excel at, but we humans can't stand, I guess that's the origin of punishments where you have to write a sentence one hundred times or copy sections from a book.

But when I think about it, if you work with computers and you're using copy-paste too frequently you're probably doing something wrong, a few examples:

1. The most trivial example: if you're coding and you've just pasted a code snippet you copied from somewhere else in your code you probably should write a method that implements this code snippet. Even if you've copied someone else's code (from the internet for example) I think that you'll have a better understanding of what the code does by retyping it and there's less chance you'll miss any adjustments you should do.

2. In my line of business I'm often tempted to copy contents of configuration files and the like but as with code, there's a good chance I'll miss some necessary environment adjustments.

3. The last example is "written in blood": you might think you see the text you're going to paste but YOU ARE WRONG. I don't know if hidden characters are to blame or some other clipboard wonder, but I keep encountering commands that malfunction when pasted: I have some OID configuration commands that do this, I encountered it when trying to install Oracle SOA Suite and just today I had a Java command copied straight from Oracle's documentation (as part of installing a connectivity web service between Microsoft's SCOM and Oracle's Grid Control) that just laughed at me until I manually retyped it.

So me, I usually retype EVERYTHING, this way I'm feeling more in control of what I'm doing.

Spooling

2010-05-18T21:03:00.003+03:00

One of the things I thought was the coolest when I was just starting to learn SQL was the idea of using SQL queries to generate scripts. Sometimes you use this ability in some kind of a GUI and sometimes you spool the output into a file which you later on run as a script (SQL, cmd, etc.). When you spool into a script file you often need to have control on the output formatting - remove "x rows selected" lines and the kind - so that you script executes as intended. To achieve this there are many SQLPlus commands like:

set feedback on\off

set heading on\off

set echo on\off

...

But it seems there's a catch, I wasn't able to determine if this is the expected behavior or just some weird bug, but a while ago we had a problem with an SQL generated script and when we tried to debug it, it seemed like none of the above commands had any effect!

(Unfortunately) I was unable to find the site that helped us to solve the case, but apparently, to function those commands need to be run as part of a script and not from the SQLPlus prompt.

Good Problems

2010-04-10T17:51:00.008+03:00

Most of my posts have something to do with complaining about different products and the ridiculous bugs they have, and anyway who loves problems? Every sysadmin would like to come to the office in the morning drink his cup of coffee (or tea, in my case) and go spend the rest of the day at the beach knowing everything will work perfectly and his presence won't be needed.

But problems, as strange as it sounds, have an upside as well.

First of all, if things were so simple, people like me would be unemployed. We'd still be able to go to the beach, but we wouldn't be getting paid for it. But even without fantasizing about a prefect world, problems are important. Might sound like I'm a bit exaggerating, but I really think problems make you a better sysadmin.

Come to think about it, problems to a sysadmin are like fights to a relationship, no one really wakes up in the morning and thinks to himself "Oh, I'd like a good argument today", but couples do fight and resolving those arguments usually strengthens the bond (and like with systems too much fighting is probably a bad sign). And like those arguments, each problem you tackle makes you understand your system a bit better, understanding that is hard to achieve when everything just works.

The ERP system I used to administer is a great example. ERPs tend to be complex systems and ours not different, we have many utility scripts that accompany the system. Sure, I knew there were some scripts responsible for printing, but until I encountered some problems with them I didn't really get to understand each and single line of code in those scripts. And on the other hand, I wrote some scripts for an interface a few years ago, and those scripts work almost perfectly (I'm not necessarily objective here) up until now, so I'm pretty sure my replacement doesn't really know what's going on inside.

So maybe I should start giving more respect and appreciation to the problems I encounter (and complain less).

Pessimist

2010-03-15T22:10:00.002+02:00

Generally, I think in most cases we all should be a bit pessimistic. Being objective is hard and personally I'd rather have a nice surprise than a great disappointment. But that's arguable - I guess optimistic people live longer - but then again, the benefit of long life is arguable as well.

But what I think is not really arguable, is that in my line of work you have to be pessimistic. A major part of being a sysadmin is keeping the system stable, and you can't really do this by being optimistic. To make the system stable you try to think about any scenario as improbable(and pessimistic) as it might be. One good example I can think of, is that having a daily backup is very important in our organization(and I really do hope we're not the only ones), yet for most systems, we've never needed those backups.

I might be taking this a bit too far when I'm berating team members who behave "too optimistically" and tend too answer "I'm getting paid for it" when asked why I'm so pessimistic, but I'm really just trying to make my point.

Fixation

2010-02-22T21:56:00.006+02:00

Being a sysadmin has many controversies, take new technologies for instance:

Recently I've installed a Weblogic server to start examining it, as our Oracle AS servers will need to be replaced sometime in the next couple of years since this line of products is going bye-bye. Anyways, it was pretty exciting, new (for me) Fusion 11g UI and the main administration screen is SO busy. So it sounds like I'm all for new technologies... not really, not always.

Yes, you should upgrade existing technologies. Yes, you should check what new technologies are out there. No, you shouldn't use a new technology where an existing one could be used.

I think the best example for my point is building solutions for specific needs.

Let's take my organization's Oracle ERP system for example. Many times a specific need arises and people suggest all kind of cool solutions. In my opinion it's very important to try and base such a solution on an existing technology, I'd rather have a solution that is not perfect than a perfect(like such things exist) solution that will handle this specific case alone and that no one will know about if something goes wrong a few years from now.

So, sometimes, I end up being this guy with a fixation on old and boring technologies...

And if talking about sysadmins, here is a good one :).

Roach-Tech

2010-01-09T14:53:00.003+02:00

When a company designs a product, there are many aspects to take into consideration. The product should be appealing to the eye, ergonomic, cheap to produce, etc.. One aspect I believe most vendors don't consider is that the product should be roach unfriendly. What am I talking about?

Well, take for example Avaya's phone:

You can see that attached to it's back (didn't find a better image) is a panel that gives it it's angle. Apparently this panel is a great habitat (at least in the Israeli climate) for roaches and they even consider it a great place to drop eggs:

That's right, this brown thing is a roach egg.

The phone doesn't look so attractive right now, does it?

Randomness

2009-12-27T19:39:00.006+02:00

Time for a break from professional stuff. The only thing common to this post and my professional posts is my tendency for voodoo crap (although I do consider myself a very reasonable guy).

Lately I've been subject to several events that made me wonder if everything is really a coincidence. What's more interesting is that those different kind of events are not something I've invented and I know many people (supporting comments please) who have experienced similar things as well (and that's not the first time I experience this kind of events as well):

1. Useful Fact - Everyone has some relatively known facts that he's missed during his life, me included. What is strange is that many times you learn one of those facts and then it suddenly pops up everywhere. My example from this month: a lecturer was giving a probability example and mentioned the - unknown to me before - fact that pingvins do not live near the northern pole, a day later this fact was mentioned in a movie.

2. Lucky Instinct - A few weeks ago I was checking out Google Dashboard and saw that my last chat conversation has been with someone I knew I didn't talk in a while. I started constructing my next blog post about how Google showed me totally wrong data, but decided to check my "All Mail" anyway. To my surprise I did have a conversation with this person. Apparently she sent me a message just as I closed GMail and unlike in other times when I miss a chat, GMail didn't put this conversation in my Inbox so I had no real way of knowing this message was sent(that's actually a GMail bug as well, but not as bad as showing totally wrong data). This person wished me a happy holiday so this coincidence saved me from being rude. This whole story is a bit like when you forget something minor at home, come back and then recall you actually forgot something much more important and start wondering what would've happened if you didn't recall about the minor thing. But here you can give credit to subconscious, unlike in my example where I just happen to check the Dashboard on the same day. On the other hand, I might have missed a lot of other chats.

3. Déjà vu month - During life you meet and interact with many people and with most of them you later lose contact, you might occasionally meet one of them but that's it. So it strikes me very unlikely that during the last month I've met\reconnected with three different people (with whom I had significant interaction before) from my past. The average amount of time I didn't have any contact with those people is 4 years! And no, those people don't have anything at all in common and don't know one another. I actually had similar month a couple of years ago (interestingly - and unlikely - I think it was the same time of the year).

Do you believe in voodoo now?

Business Model

2009-12-12T11:04:00.002+02:00

Every company has a business model. It can focus on selling a product, selling accessories for the product, selling support, etc.

One type of model is the "consultation" model, it's used when the product the company sells is sold for near to nothing, but the company still has to make money somehow, so the company presents you with experts that will help you through every phase of the implementation.

Now there are two options - the consultant gets paid either on a per project basis or on a per hour basis. The per project option is OK, but it's highly unprofitable for the company - it's never a good idea to commit yourself to an amount of work you don't know how to translate into time(==money).

The per hour option is problematic from the client's side. A "good" consultant from the company's perspective is one that can do the same work for the longest amount of time - to me it seems to contradict professionalism, but that's not my point here - that kind of consultant has no motivation to do his work to the point and as effective as possible or even answer questions on the phone - every question can be seen as something that should be thoroughly discussed on the client's site (and expense).

Now, with support things are easier - if a client feels dealing with a support issue is taking too long he concludes there's a problem with the company, and that's obviously not so good for the company. But consulting taking a long time can easily be painted like a good and thorough work on behalf of the consultant, and since the client often really doesn't have a good idea about the product he can be easily fooled.

Another pitfall you should beware of when taking on a software project.

High-End Software

2009-10-30T16:27:00.009+02:00

Like I mentioned a few months ago, I'm currently working on an Enterprise Search project. Since then I discovered a few other annoying things about FAST ESP and other high-end search solutions.

I've tested two main types of search solutions:

1. Entry level solutions - by companies that have another core business except search.

2. High-end solutions - by companies whose core business is search.

Obviously, there a lot of products somewhere in between, but that's always true. Actually, FAST has been lately acquired by Microsoft, but for the purpose of this post it is still under the second category.

Being a newcomer to this field of Enterprise Search I was quite innocent and thought that entry level solutions will be simple, basic and easy to use - no disappointment here, but I also expected high-end solutions to be a complete search suite that can do great and cool stuff.

Well, products I've examined can do great and cool stuff, I'll even exaggerate and say that it feels like everything is possible if you know your way around the product, but in no way these products I've examined are a suite. It almost seems (it's actually sounds like a swell business model) that the products are intentionally designed to make you grate your teeth at every step so they can provide their business partners with work for their consultants. It seems like they try to make the product as naked as possible, leaving only the basics of indexing efficiently, providing customizing tools and a few other abilities. As always, a whining is not complete without a few examples. I'll rely mostly on FAST ESP examples, but not only, I just know it the best:

1. I would expect a high-end solution to include some security enforcement. Say I want to index a file system(obviously there are other examples as well) content source, to the entry level products it's obvious that I want to index the ACLs as well, not so much to the high-end stuff. High-end software will require installing an additional module that I'll have to carefully configure. And that leads me straight to the next point...

2. To configure the security model for FAST ESP to enforce file system security I have to follow the documentation which, put in one simple word, sucks.

Fact: surprisingly the FAST guys realized I might want to index Windows based file systems.

What I'd expect to see in the documentation: a simple to-do list containing every step I should do in order to configure the whole thing.

What I actually got: every piece of the puzzle is in a different part of the documentation so before each of the following revelations I had to wonder why nothing works and why nothing is written (or at least not where I expect to find it):

When the module is initially installed no indexed items are searchable, there are a few ways to work around it and they're written at the end of the documentation.
There's no built in authentication module (a pity) but at least there are a few ways to work around it as well.
To index securely you have to modify the processing pipeline (or the whatever the term the product uses), next point here I come.

3. The products I've examined have far too much of encoding related (Hebrew and computers should not co-exist) issues, some of them I've been able to overcome but still, for now I can't index securely paths with Hebrew in FAST ESP.

I have lots of other examples and I'll probably have a lot more as I go along, but I think I made my point. Totally high-end...

Development

2009-10-21T20:31:00.006+02:00

I'm not a developer but as the team leader of the sysadmins (which is often referred to as the "technological team") in my department I am interested in the platforms and methods that are used for development, after all the bottom line is that I'm responsible for the stability of the systems and a system is more stable when the correct practices are used.

Let's take for example our Oracle ERP system (which I used to administer so I'm more familiar with it than with other systems). Today we have web applications based on different technologies:

1. PL/SQL cartridge based applications - well, the programmers do know PL/SQL so it's kinda OK, but no new applications are written using this technology and in the next EBS version (R12) it's no longer supported.

2. Java based applications - no one really knows them and they're supposed to be re-written.

3. ApEx based applications - small modifications are still made to them but the knowledge level is pretty low.

4. .NET based applications - that's the technology that all the developers are familiar with and with which the latest applications were developed.

We have all those techniques since each of them fitted a different need at a different time.

One technology we don't currently use is the Oracle Applications Framework which is a pretty strong technology but...

We currently have about 3.5 EBS developers none of which is really specialized in Java (and the Framework is Java based) and the turnover rate in the developers' team is pretty high.

These facts don't really get along well with the multiple development techniques we maintain.

So now it seems like the time to make a strategic decision as to how the EBS development will look in the next few years, I can think of two main approaches:

1. We choose one single technology that we'll use to write all the solutions, obviously it won't meet perfectly all the needs but the programmers will be specialized in this technology and training will be relatively simple.

2. We choose two technologies that will answer two major types of requirements - for example general user applications vs. specialized user applications. The programmers will be less specialized but we'll be able to provide better solutions.

Anything more than that will have too much training and maintenance overhead...

Disappointment

2009-07-17T10:01:00.003+03:00

Sometimes you think you've just seen something amazing: on news, a discount or some great technology, since this blog has a certain orientation, I'll focus on the later.

This week I was planning my oncoming trip to Slovenia, and me and the Doc got to the point of looking at transportation, specifically trains.

Google led us to this site. If you enter the site you'll notice it adapts to your localization settings - nice! We've started typing "Venice" in the destination and not only we had auto completion we saw support for different writing options for "Venice" - really nice comparing to the dozens of other transportation sites we've seen (at some point we thought it could read Hebrew but that's really an exaggerated expectation). And now to the most amazing feature (tam tam tam...) the prices appear in local (ILS) coin! After being amazed for a couple of seconds (yes I know, it's not really that a sophisticated technology, but as mentioned above we did see other transportation sites), we've decided to check what the British version looks like.

Apparently the two sites give different prices for the exact same trip. Yeah, that's right, three adults traveling from Ljubljana to Venice buying tickets in Israel will pay 453 ILS and the same three adults will pay only 42 pounds traveling from Great Britain. European discount? Don't think so, since Australians pay less. I've tried long and hard to think what could be the code behind those ridiculous results, but I still have no idea, suggestions?

So, going back to the first paragraph the conclusion is that miracles do not exist, sorry kids.

Making Me Sweat

2009-06-20T19:34:00.003+03:00

For quite some time now, I've been working on an Enterprise Search project for my organization. Last week I've installed a FAST ESP system as part of a POC, the system uses many third party software components and since each has it's own license they can't be provided as part of the ESP software and have to be downloaded as part of the installation process.

But, here's the catch, I don't have any Internet access on my network. The installation guide provides a solution for this kind of scenario - install the ESP on a computer with Internet access, after the components are downloaded you can abort the installation and transfer the components any way you want to your LAN.

This is a very strange solution, why run an unnecessary installation? What if I have limited privileges on my Internet computer?

After all, there's a much simpler solution - why not provide me with the links to the components? After all, what I did was to harvest the links from the manifest.xml the installation uses to download the components...

Why make me sweat?

Storming In Or Taking Baby Steps

2009-06-08T21:34:00.002+03:00

When you have a problem (specifically, technology related) there are two main methods for dealing with it:

1. Storming In - throwing everything you have at the issue, e.g.: trying every performance related command you know, changing hardware, etc.

2. Taking Baby Steps - slowly and thoroughly analyzing the issue, e.g.: reproducing the issue on a development environment, checking all changes that have been made, etc.

Reality, like always, is some shade of gray so what you do in practice is usually some combination of the above two methods.

Many times it's not clear what approach to use. Storming in will sometimes solve the current crisis but might handle only the symptoms leaving you unprepared for the next time the problem manifests itself (and this time the instant solution might not work). Generally, it's always better to understand the causes for a problem and to investigate them, but sometimes issues won't reproduce on development environments and the causes are just too voodooish and some issues are just simpler to ignore. And after all, time IS a valuable resource.

The Stage I Fear The Most

2009-05-28T19:03:00.003+03:00

Since I became a team leader I've been working more and more with Oracle products other than EBS, mainly iAS and OID. I've noticed that the installation part of iAS-like installations never fails, the stage I've really learned to hate is the Configuration Assistants - each one that completes gives me a sigh of relief, but too many times I'm cursing instead.

For instance, lately I've been trying to upgrade my 10.2.0.2 OID instance to the latest 10.1.4.3 version, upgrade path looks like this: 10.2.0.2->10.1.4.0.1->upgrade MR->10.1.4.3.

10.1.4.0.1 - check. upgrade MR - check. 10.1.4.3 - uh oh...

Even before installation begins, Oracle tries to outsmart me - documentation says to shut down OPMN, but when I do that I get an error during the initial steps that says it can't determine running processes(like daaaa!). OK, so I start OPMN, but then I get a "dude, you're running services that bother me" message, "OK take it easy, I'm shutting it down" is my reply. Up until now I have the upper hand and the installation itself runs smoothly, as always.

But then comes the one before last CA - the DCM CA that fails because of a problem with an Apache dll. So after trying to solve it for quite some time (replaced dlls, configuration files, etc.) without any positive results I opened an SR, the action plan , is kinda funny - install 10.1.2.3 on top of 10.1.4.0.1, then try again, makes sense don't you think? Well, it does (a bit), my previous attempt did include a 10.1.2.3 upgrade before 10.1.4.01, some maybe I should've thought about it myself.

So I did just that, but then the SSO CA took ages (more like two hours, but you get the idea) to fail, some mixture of opmn stopall/startall got it through but now the OPMN startall CA fails so I didn't even got to the problematic DCM CA yet, I guess I have some frustrating days ahead...

With the CAs I always have the urge to just skip 'em all and run 'em later, on the other hand if things are wrong with the so basic OPMN startall maybe I should handle things now. I'll probably have more updates regarding this issue in following posts.

Linguistics Tip

2009-05-02T11:02:00.005+03:00

OK, so we all know that when facing language issues, playing with NLS_LANG often does the trick, but a while ago we had an issue with a web app (running on iAS) that would show normally Hebrew characters for strings taken from configuration files but would transform any Hebrew data from the database into question marks. The interesting thing was that the same application running from a workstation worked perfectly.

We were starting to get desperate, as more and more people looked at it and didn't manage to find a solution. But when I showed it to the DBA in my team he solved it in five minutes, luckily, he had a similar issue the week before.

Here's the trick, sometimes you have to modify the language settings for the server itself. Actually it's not enough to change the language settings, you have to check this annoying check box as well:

We've obviously played with the Regional Settings before just never even looked at this tab and hence didn't see any changes.

Linguistics

2009-04-24T00:23:00.003+03:00

My working assumption always was that about 33% of my duty as an administrator is language related, it might sound like a hell of an exaggeration but experience shows I'm not that far off target. You can always count on Hebrew to supply some issue with printing, badly encoded databases or question marks instead of readable text. The problem with linguistics' issues is that they tend to be hard to solve, you would expect most of these problems to have a common root cause but the fact is that a new solution is required each time, you do collect useful tips as time goes by but there's always some new issue waiting around the corner.

Hack, if I think about what I did for the last month it seems like most of the time spent was somehow connected to the fact that English is not the only existing language, I even think I have material for a couple of posts with useful tips and a lot of desperation.

Usually when I encounter these issues I blame Hebrew for being so odd - wrong writing direction and weird characters don't make my job easier, but English seems to be pretty fucked up as well. Sometimes I wish I could find the guy who invented case (as in lower case and upper case), it really makes no sense, OK so we'd write "english" instead of "English", who cares!

About a year ago I've mentioned researching EUS, well, we managed to implement (the rather cool) design we wanted but not without trouble.

As part of the design we wanted to run command line esm commands to assign global roles to enterprise roles, the problem was that whenever we added a global role with the command line tool the whole list of global roles would just disappear from the GUI. How come you ask, and case sensitivity I answer! The issue was that the record inserted into the OID had case issues. It took me a few months to stop waiting for Oracle's solution and a few days to decompile the code and find the problem, it took Oracle another couple of months to supply a working patch.

And all of the this because of that stupid case invention.

Security Issues

2009-04-12T15:55:00.007+03:00

A while ago I was phoned by the Apps DBA when I was away from the office and he told me the production is experiencing an issue that includes concurrents failing with an error message regarding access problems to .tmp files. It's not the first time I encounter this kind of problems and it's needles to say that it was never the stated issue with file permissions or disk space. So I came to the office to help the DBA to solve this issue (nothing trivial worked).

Solving the problem was odd, I had a good guess regarding the solution based on the previous cases, but it's really strange it did work since if what I did fixed a real problem I have no idea what triggered it and how the issue didn't manifest itself before, since nothing I know of changed (definitely not the problematic database parameter). Concurrents just started crashing with no apparent reason.

But that's not the main issue here (unless you're stuck with it and then you better know the solution or else you'll have some "fun" trying to solve the issue).

The really alarming thing about this case is that when concurrents crashed some of them had the apps password (in cleartext) in their log files - the same log files every user can see for the concurrents they submit.

Really, I couldn't believe my eyes, it seemed like the format usually present in those damn .tmp files, but I really don't understand what kind of a leak can cause the apps password to be dumped into the log file, to tell the truth it almost looks like an intended (buffer overflow?) attack.

The implications are very unnerving, what if we didn't happen to check a problematic log (not all of them had the issue) or what if we had just overlooked it? The meaning would be that everyone would be able to see the password until those logs were purged and we wouldn't have a clue. How can we be sure it won't happen again and who guarantees we will notice it the next time?

Thinking With A Pen

2009-02-13T17:31:00.002+02:00

Those of you who read previous posts of mine know that I love the connection between the human brain and computer science. One of the topics I've learned during my Neural Networks course in my first degree was Associative Memory, and indeed I'm often amazed at how memories and thinking in general are association driven.

I may use all kind of technologies to manage my data but I still always keep a primitive notebook+pen set on my desk. You can say that there are things that are just more easy to do with a notebook, and that's true, but in addition, this notebook helps me to think. I sometimes only need to grab my notebook and pen without any scribbling and I already get into "thinking mode", it's like the pretence of writing signals to my brain that it's time to work.

When I think about it, I actually can't perform heavy thinking without a pen in my hand. I remember last year when I had an Advanced Algorithms course, the homework weren't so hard so most of the questions I solved with a pen and a blank paper in my hand, whenever I tried to give up the pen it just felt awkward.

Another example is that I can't remember some fact I always try to think of a situation in which I used this fact and it often helps (in other cases I just can't remember no matter what, but that's another issue).

Trust No One

2009-01-02T17:49:00.002+02:00

One of the TV shows I like the most is House. As many people point out, each episode is pretty much the same and yet it's one of very few shows that improve as seasons go on. Anyways, one motif that appears in almost every episode is that people shouldn't be trusted under any circumstances, every patient House treats has some terrible secret lurking in the dark.

Users are sometimes like patients, they don't necessarily have a dark secret (although they can occasionally forget to tell you an important detail) but they definitely cannot be trusted. That's why the very first step of handling a problem is watching it reproduce with your own eyes, there are a couple of reasons I can think of for doing so: first there's the Sysadmin Effect - never underestimate it. And then there's the fact that the reported problem is often not the issue at all (or not at all an issue). Examples I've heard of and experienced myself range from "The system is down" when working on a computer disconnected from electricity to calling hysterically about "a critical application is down" when the application is neither important, nor down. Obviously there are more examples in the mid-range but that's just a blog post not a thick book.

I have a special place in my heart for the guys that have some good advice, like the totally non-technical guy that advised me to add some memory to my servers to improve performance or the one that had an idea about how to fix the application (the OID) when the issue was with his OS.

So remember, always begin with the first step.

Simple Testing

2008-12-26T17:05:00.007+02:00

A while ago I've upgraded my whole development AS infrastructure to 10.1.2.3, after the upgrade SSO stopped working. Since SSO (at least mine) tends to be a bit fragile, I sighed with a familiar feeling that this is going to take some time and started browsing through the logs. Once again Oracle proved that their logs suck and tend to display the same error message for totally different errors. So the logs showed an error that usually accompanies a bad keytab file, since I had some similar issues lately I've decided to re-create the keytab file, but it didn't really help. Apparently, all I had to do is to search the Metalink.

The real issue is that the jdk version this version(actually not only this one, if I'm not mistaken IDM 10.1.4.0.1 is as well) is shipped with (1.4.2_14) has some error that prevents SSO from working, the solution is a simple one - install a higher version (say 1.4.2_19).

Now, these things get me really frustrated. OK, I get it, your error handling is not the thing you take pride in (sure hope not), but it's not the first time (more examples to come in following posts or you can read this post again [last example]) it seems nobody have really tested the final product. Had somebody taken the final product (with jdk 1.4.2_14) about to be published for everyone to download, installed it and tested it for the very basic functionality this error would've been discovered (and hopefully the product wouldn't be published).

That's the part when I start to imagine the following conversation (a special bonus for whoever discovers the meaning behind the aliases):

M: "Hey, there's a new jdk out! We should ship 10.1.2.3 with it because it's the newest" (and new is good, right?)

S: "But we tested it with 1.4.2_x<14!"

M: "Yeah man, but we didn't really touch something that heavily depends on jdk specifics, it's a minor version anyway."

S: "You know what, you're totally right, let's do it! It's not like we've ever shipped any totally-unworking piece of code before."

Really, with the simple applications we have in my company we try to test them thoroughly and even though not always successful try to enforce different rules before deploying, things like a certain period of time in which the application has to work on a test environment without code modifications before moving to production, a clean testing environment for relatively big installations, etc..

So how come Oracle manages to ship a totally not working version of a product?

Noise

2008-12-18T18:14:00.003+02:00

Well, I haven't blogged for a while, you can probably blame halo 3 for that as well as some other XBOX games. Anyway this post is not going to be too professional as I just want to raise an issue that deeply bothers me.

Lately I have been to several meetings in different companies, at most times I haven't been too satisfied with the hosting. How come?

Well, there's this issue with cookies and other snacks placed on the table, they tend to be noisy - I mean every bite you take makes a sound so loud it reminds me of the reasons I haven't blogged for a while, if you know what I mean. And so instead of enjoying a snack I end up trying to silence the breaking sounds - can't people provide snacks that behave?

In one company they gave me a drink in a glass bottle, the problem is that the table in the conference room was made of glass as well, so putting the bottle back on the table after drinking without generating a loud response required great skill - I kinda started imagining me as a Terminator measuring the distance to the table and the amount of force needed to put the bottle back silently.

Please people, think of me next time...

Digging

2008-11-08T22:43:00.003+02:00

One of the abilities a sysadmin must have is the ability to dig. What do I mean when I say "dig"? Well, I mean that many times you face a problem that seems to be rooted deeply inside the system, and a sysadmin has to know how to solve these problems. The first thing I personally do most of the times is to say "OK, that's as far as I'm going, time for official support", but as I wrote before, this usually ends up with me de-compiling Oracle's code.

Time for an example.

I've recently been trying to utilize Oracle's ESM (Enterprise Security Manager), since the system it's intended for requires batch actions I used the command line tool. Guess what, when I use one of the commands the GUI stops showing associated roles for the related Enterprise Role, since it was obviously a coding bug, I've logged an SR thinking to myself that this bug, if reproducible (and it is) means no one have ever tested this functionality. About a month later I was notified that a fix was created and they're testing it. Two days later - bad news, the fix did not pass testing. And this is actually the current status (a few months have passed by now).

Last week I got tired of this whole thing so I've started digging for the source of the problem. Apparently, when you use the command line, the record generated in the OID has a case mistake. So my next move was to de-compile Oracle's code and fix it, and voila, it works!

When I told the support analyst about my fix, he said he knows the solution is easy but the problem is that they have some incompatibility issue with different database versions. I really don't care if it's reasonable for such a small fix to be taking such a long time to implement, I'm just happy I know how to dig.

Life-Work Balance

2008-10-31T10:41:00.003+02:00

It doesn't matter how intensive your work is, how time consuming your studies are and how tired you get home after all of them. You should always make time for some other activity, it can be sports, some artistic hobby or whatever, as long as you enjoy it. Another important thing about this hobby is that it must be something you do constantly and don't skip no matter what. It might sound complicated but it's actually as simple as forcing it into your schedule and not giving up on it (and to yourself).

For me, practicing Judo is a great way to get my mind set on something other than work and everyday thoughts, it makes me concentrate on an entirely different thing. One of the things I like about it is that the great physical effort you have to put into a training session is a huge contrast to my work, that involves mostly sitting in front of a computer in different angles.

Yes, it can get tedious, making progress is hard and there are those days when I have to struggle myself not to give up on the training session. But the fact is that most of the time and especially after showing up for a session after missing a few I have real, genuine fun and that's what it's all about.

Traveling

2008-10-14T15:46:00.005+02:00

I haven't blogged in a month, and for a good reason too. I was traveling Ireland for two and a half weeks and after physically landing it took some time to mentally land and even more time to "catch up with life" - organize my photos from Ireland, watch all the episodes I've missed while being abroad, etc..

This trip is obviously something I should write about, my trip companions actually offered all kind of ideas during the trip, but I don't really know what to write. As I write those lines I don't know yet what I will write about. Really. I guess I'll just write some outline of the trip and interesting highlights for those of you who plan to visit Ireland as well.

Part I - Shortened version of the Wicklow Way

Since we wanted to make the most of the time we had and the last part seemed a bit boring, we decided to start from Dublin and walk only for five days - up to Moyne/Tinahely and not up to Clonegal.

The walk was really great. Especially since we had no rain at all during walking (and in general great weather), actually we had lots of luck since everyone we talked with before the trip told us we'd be totally soaked inside out, in reality the massive coat I brought ended up as just a burden, the only times I took it out of my bag was when I moved it to another bag. We had some rain during the first two evenings and some rain on the last two days in Ireland - and that's it. Oh, and one evening when one of the guys hung his clothes to dry outside, but that wasn't me so it doesn't count. All in all this absence of rain is another proof to my belief that I have tons of excess luck. Anyway, words (except maybe "Green") won't really describe our walk so you can either check out the photos or walk the Wicklow Way yourself (recommended option), and we've learned a few lessons as well:

Tip 1: Public transportation in Ireland sucks:

a. It's expansive - buses are expansive, taxis are expansive and trains I prefer to forget (31.5 euro for a train from Galway to Dublin).

b. It's hard to find. When planning to quit the Wicklow Way early we didn't consider the fact that we won't be able to find a bus and the "taxi" we eventually used was an elderly guy with a van (that robbed us nonetheless).

So what do you do? Whenever possible just rent a car, this method has it's own limitations (drop-off points and the like) but it's much more convenient and cheaper.

Tip 2: Don't carry around your big bags. The places you stay at, will be glad to transfer your belongings to your next stop for a fee (usually 20 euros), but it's worth it. And if you picked Ireland you can probably afford it.

Funny 1: We're lame. During the walk we've met many people doing the Wicklow Way. Many of those people aged more than twice our own age and they didn't even bother listening to my second tip... embarrassment indeed.

Part II - The South

Next we rented a car and visited:

a. Around Cork:

Blarney Castle - a nice castle which has the "Stone of Eloquence" on the top floor, kissing it is not a simple task, so unlike other visitors we decided to pass.

Fota Wildlife Park - a very nice wildlife park - unlike a zoo, animals are not behind bars and mostly (not the cheetahs though) are separated from humans only by water or modest wire fences.

b. Around Killarney:

A whole day of cycling in the Killarney National Park - Ross Castle, the Gap of Dunloe andMuckross House, including sailing through the three lakes by boat.

This little adventure took some recovering from.

Part III - The West

In this part we dropped of the car and stayed for three days in Doolin - the place where everyone owns a B&B (at least one).

a. Walked the section of the Burren Way from it's beginning to Doolin (we took a bus to the beginning) through the magnificent Cliffs of Moher.

Tip 3: The Burren Way. Looking now (we didn't have a map as we did the walk) at maps I have, it seems there are at least two versions of the Burren Way - one that starts in Lahinch and simply skips the Cliffs of Moher and one that starts at Lisacnnor and goes through the cliffs. We tried to do the second version but it was very poorly signed (unlike the Wicklow Way), luckily we realized going down at the cliffs didn't make sense.

b. Took a ferry to the largest of the Aran Islands - Inishmor - and cycled through it (this time it was much easier).

Tip 4: Unless you stay for the night you have only about three hours to visit the island before the return ferry arrives.

Tip 5: There's an option to take ferries for the islands from Galway as well, an option worth exploring.

c. Spent about half a day in Galway.

Part IV - Northern Ireland and around

After a brief stop in Dublin to celebrate Rosh-HaShana and rent a car we headed up north.

a. On the way north we visited the megalithic tombs at Bru na Boinne.

b. We stayed at Carlingford near the invisible border and did a short (about 11 km) walking route - poor signing again.

c. Drove to the very north of Northern Ireland and been to the Giant's Causeway.

Funny 2: When we finished with the Giant's Causeway it was already dark, we were hungry and we had a long long drive back to Carlingford so we entered a nearby hotel to eat. I don't know why but we were total strangers there - everyone looked at us, they didn't know how to charge us for the beer and asked us to another room to serve us dinner, strange...

Part V - Dublin

We had about two days in Dublin - one before going up north and the last day, can't say the city is too exciting, the Guinness Storehouse is nice but unless you're a Guinness fanatic it's not that much of a thrill. Also there are some nice museums.

That's all folks.

Of course, if anyone is really planning on visiting Ireland, I'll be more than happy to answer any questions.

Buffering...

2008-09-15T23:25:00.002+03:00

It's not really clear what I'm doing writing this post when I'm leaving for Ireland in an hour and a half, but I guess I have a buffer to waste...

Anyway, this post is about the single most important part of any IT (any?) project - the BUFFER. A big enough buffer should always be taken for whatever procedure you're planning - we usually take a 30% buffer.

I'm not talking only about an installation process, but also about things like giving an answer to your manager as to when you'll have a new product checked or a patch installed for the first time on a development environment. I'd prefer to get a raised eyebrow and maybe surprise my manager later to working under unneccesary pressure and making up execuses. Same goes for actual installation processes, I'd prefer announcing a long downtime and getting complaints from everyone to trying to decide in a rush if I should rollback now or hope I'm lucky enough to shrink an hour's work into fifteen minutes.

Well, that was my little - written in blood - piece of advice before I drown myself in beer, whiskey and god knows what else...

Ignoring My Issues

2008-09-06T12:28:00.002+03:00

A big portion of being a sysadmin is solving problems. One of the first changes I've noticed when I stopped being a sysadmin is the fact that I suddenly have (much) more time on my hands, time previously consumed by solving - usually minor - issues. But I think a sysadmin's job is also to know when to walk away from solving an issue, let me give you a couple examples:

There's a client side issue with the EBS system I used to administrate, I was never able to reproduce it at will or find it's cause, but I do know that reseting the Windows profile for the client solves the issue every time. So that was my policy, I've decided that if the workaround is so simple and finding a real solution is so complicated (believe me, I've tried), I'd rather just ignore the problem.
Last week the new EBS sysadmin was trying to solve an issue with one of the EBS related custom applications we have, it has suddenly stopped working without any apparent reason. This application resides on an Oracle Application Server so he was going over the logs trying to find the problem. When I saw it's taking too much time I came over and just restarted the Process Manager and voilla, it works. Talking to him afterwards he said he was aware of this solution but as it was past working hours he thought he'd better explore the issue - well he had a point. But my point is that an issue that happens twice a year or so and that can be simply "killed" isn't worth wasting your time on even if "killing" means affecting some other applications running from the same AS as well.

So how do you know when to ignore a problem? There's no real answer for that but you should probably consider the following aspects: Is there a workaround? How bad is it? What's the frequency? How long to work out a solution?

Upgrading + Blog Day

2008-08-31T20:17:00.004+03:00

This time I won't be telling about a database upgrade.
Just wanted to tell that from tomorrow on I'm upgrading myself. From tomorrow on I'm no longer an Apps DBA, my replacement has completed his training and is ready to take over the job. Effectively, he's kinda been doing 99% of the tasks for at least two weeks now.
So now I'm a team leader of a team of sysadmins - the team I was previously a part of - except from an EBS sysadmin the team also includes other sysadmins (MOSS, DBA, etc.). As a consequence the emphasis in my posts will shift a bit from EBS to more general stuff, but there'll probably still be some amount of posts about Oracle products.

And to the second part of this post.
Didn't know about Blog Day until today, but since I'm a happy victim of it's customs, I feel obliged to "celebrate" it as well, so here's my list:

techno-mosh. Mosh is a very versatile guy and has many areas of interest so it's always fascinating to read his blog. Besides, he frequently writes about all kind of great tips (that get him fans from all around the internet :)), so it's definitely a most recommended blog.
Coding Horror. Definitely a great programming blog, always gives me insights and even if not it's still fun to read.
Steven Chan's Blog. Although, as mentioned, I'm no longer an Apps DBA this is still a great blog, that many times gave me new ideas and new features to research.
xkcd.com. Great (if somewhat twisted) humor.
Yevgeny. Gotta agree with Mosh on this one, Yevegeny does a great job sorting out the less interesting posts and sharing only what I want to read.

Happy Blog Day everyone!

Bad Design

2008-08-16T19:46:00.006+03:00

Yesterday I was shopping for clothes.
Usually (at least in Israel) dressing booths consist of a small booth that has some hooks for hanging clothes and that's all, not fancy but good enough. One of the stores had a somewhat different design, it's booths didn't have hooks and had a kind of a bench instead and it had mirrors on each booth wall as well - not full size mirrors but a pretty big round mirror on the wall opposing the door and stripe like mirrors on the other walls. It's probably not the only place they have those mirrors but I find this idea pretty bad. What are the mirrors for? Usually the mirrors are just outside the booths so the shop assistants can do their job - tell you how great you look in this shirt and that they have just the perfect pair of pants to go along. If I just try on everything inside the booth isn't it a missed opportunity?

Well, actually in this specific shop even when I used the mirrors outside the booth (the ones inside the booth just weren't comfortable) the shop assistants didn't do too good a job. Instead of a casual "Can I help you?" or "What are you looking for?" I got an awkward "Would you like to get service?" (from all of them so it's oughta be some kind of weird policy) or even worse - for Hebrew speakers - "אתה רוצה לקבל שירות ממי?". In addition to that, I could barely understand what one of the shop assistants was saying(mumbling actually), don't they check for such things when hiring people for this shop?
I wonder why the bad service, I mean, other shops in the same mall have normal shopping assistants. Do they pay less? Are they less thorough in choosing their staff? They're lucky I love their clothes...

adpatch VS. OPatch

2008-08-14T23:21:00.003+03:00

So, in my previous post I've introduced some issues with both patching tools, but the truth is that adpatch is usually pretty good. Usage is pretty trivial, you have some useful flags (I often use -nocompiledb and -prereq), logs are pretty coherent and you have a good control over the workers with adctrl - although I don't really understand why the so useful "Skip" is hidden under the 8th non-visible option.

OPatch, on the other hand, I really don't like.
How come the architects behind adpatch thought about storing all the version information in the database and the architects of the database itslef didn't come up with this same idea? Why do I need to maintain the inventory folders? It's another thing to take care of when cloning environments, actually, in my case I just ignore it - I don't even know how to clone it properly (although I'm pretty sure there's some documented procedure for this).
Why when I wanted to upgrade my 10.2.0.3 patchset X database to 10.2.0.4 I had first to remove the patchset? And why when I tried to do this I got wierd OPatch errors? Isn't the whole idea of a patching mechanism is to save me those kind of trouble?
It seems that OPatch is nice for checking what you have installed on your database, but as soon as it comes to installing any patch beyond the first one it's nothing but trouble.

So ok, when two patched make different modifications to oracle.exe it's a problem to resolve the conflict, but I refuse to bilieve it can't be done.

Flexibility

2008-08-07T22:40:00.002+03:00

Patching mechanisms such as Oracle's opatch and it's more successful brother adpatch (this issue deserves a post of it's own) are great stuff - you can introduce specific bug fixes into the system, install a whole new functionality or make your application compatible with a new database version and all without having to upgrade a whole application version. The other side of this coin is the fact that you can ignore bugs you don't care about and you don't have to maintain and debug functionallity you don't use. Well, sounds great, everyone gets a suit sewn to perfectly (or at least nearly perfectly) match his needs.

But those mechanisms frequently require strict dependency and versioning management, and sometimes, I think, Oracle just exaggerates.

Let me show you what I'm talking about. During installation of SCM_PF.J on 11.5.9 I got many failed workers, two of them relate to my point:

1. A worker failed and searching Metalink I've discovered that this bug can be fixed by applying a database patch. The README file for the patch says that I should apply the patch and run drkorean.sql, it also specifies very strictly that I should make sure my database is down during the process. Off I go and try to install the patch, but opatch complains about bad inventory which might actually be true since this is a replecated development environment and it might have a problematic inventory. But I'm not the one to give up easily, so I take a look at the actions.xml file of the patch, and... big surprise (not really)! All the patch does is to copy drkorean.sql to a designated folder. A more naive sysadmin could've started panicing, trying to fix his inventory or even open an SR to cure the poor thing.
Wouldn't a simple "copy-run" instruction be a better option? It's not like there're problematic versioning issues here - it's a single sql script that fixes a specific bug - if you insist on having some supervision here, let me take care of this single script's version, believe me I'm up to it. And what's that about taking my database down? Guys, I was in the middle of applying a patch!

2. Another worker failed and Metalink pointed me to a "PROBLEM" document that said that I should apply a patch - unluckily the patch is for 11.5.10, no good for me. Like in the previous case a brief glance inside the .drv file proved my suspicions that all the script really does is to replace the sql script that the worker failed on with a newer version. I'm 100% sure that there's no problem running it on 11.5.9 - and that's actually what I did.
OK, I get it, the bug was originally discovered on 11.5.10 and that's probably the only setup it was ever tested on. But, wouldn't arriving at the conclusion that it matches 11.5.9 as well save us all some time and blog lines? Woudln't it be ashame if I had to open an SR and wait for a reply to handle this trivial case?

Thank God for Internet

2008-08-02T15:33:00.003+03:00

I sometimes think that the Internet - with emphasis on the ability to search it - to technology is something like zero ('0') to mathematics, it's a relatively late development and without it some things are just impossible to accomplish.

I've recently been working on some small C# project. I don't really know C#, yeah I did a one-week course and I do know Java and C++, but that doesn't really mean I know all the techniques and tricks. So, naturally each time I hit a bump, I google it, and voila! The first or second result is usually what I'm looking for.
Now, what would I do in a world without Internet? Call a friend? Look up the subject in a C# book? This will probably force me to keep a small library of programming books, which is nice to brag about but space and money consuming and still much less efficient. And neither solution allows me to instantly paste code snippets into my code. You get the idea, not having Internet, my task would have been much more frustrating and challenging.
But who cares about me, what about serious software development? Think how much time would've been wasted if every time a programmer had a problem he would've had to start searching books. Of course he has his colleagues as well but they don't usually provide a written code example, more wasted time. And time is money.
I'm not really a (programming-)linguist but I think that modern programming languages like Java, C#, Perl, etc. couldn't be what they are today without the Internet, they depend on too much different interfaces and classes (hey, where would I get my Perl packages from) and one can't expect to master them all, there's also a limit on how much information and examples you can put on an MSDN disk you even if you release a new version once in a while. Besides, you'll never have what professional forums supply.

Since I work mostly with Oracle products, the trivial analogy is Metalink. Yes, I know Oracle precedes the Internet but I really don't know how support worked those days. Did they publish some kind of soft copy repository containg notes? Did you manage all your TARs on the phone? And what about uploading log files, did you have to send a diskette with the files to Mohender in India?
One thing I'm (pretty) sure of is that having an online support system, like Metalink - however not efficient it may be at times - enables Oracle (and every other software company as well) to develop more complicated products and more sophisticated features. It's one thing to search the Metalink or google once in a while and a totally different thing to search through documents and manage your TARs by phone.

Hail the Internet!

Mass Changing

2008-07-24T23:40:00.002+03:00

In some cases the domain name for an entire Oracle product has to be changed. Oracle tools have a pretty straightforward approach to this task: replace all occurrences of the domain name in all configuration files. Now, that's ok if the source domain name is lribinik.cool.blog, but what if I'm more enamored with Oracle than with myself and I happen to have a single label domain named ORA? Things are gonna get messy in this case (just use your imagination).

One of those cases is when you perform a cloning of your EBS environment and run adcfgclone. Now, I used to have this issue forever but I only noticed it used to target files under rdbms\admin, but after installng 11.5.9 CU2 (I think this is the patch to blame, but I might be wrong) I noticed every possible location is trampled by the huge Oracle monster: perl, jdk, jre, you name it. What da hell?! Why should perl\java code be changed AT ALL after a cloning process?
Had enough of this, really. To deal with this I've just removed lines relevant to those folders from the instconf.drv file and now I can clone quitely.

Similar processes for iAS domain name change are a bit more reasonable, they actually warn you about this in the documentation and let you know about a file that contains the list of files that will be changed, they actually try to stick with to relevant files.
I'm not sure though, if the chgip script (the one that changes the domain name) works with this file, because I think files not mentioned there are changed as well.

Anyway, I think the best way to avoid this problem is just to name your domain lribinik.cool.blog. It's a nice name.

Education, Part II: Exams

2008-07-17T23:09:00.003+03:00

I think I might have made a wrong impression on my previous post, studying is not really such a nightmare to me, I do enjoy it, it's just that sometimes it can get very exhausting and frustrating.

One of the aspects of studying that leave me emotion-mixed are exams.
On one hand, the exams' period is usually the period during the semester when I have the most free time - no lessons, (usually) no assignments and as my studying method is a quick revision of the lecture notes and solving previous tests I don't spend much time studying for an exam. Why not? Well, at least in my Masters I usually have a maximum of two previous exams to refer to and this doesn't leave me with much studying to do.
On the other hand, exams are annoying. One of the things that annoy me is that you would expect an exam to be designed to test your knowledge, but this is not always the case. That's why my studying method is as described above. You are not studying the course's material, you are studying for the exam.
For instance, I had three tests this last semester, each one of a different kind:
1. "A Literature exam", or at least this is the Hebrew term, meaning the exam isn't really designed to test your knowledge or understanding but how much you can write on a trivial subject and how accurately you can guess what negligible facts the lecturer expects you to mention.
2. An easy test but there was a catch... time shortage, and the lecturer won't add any time. So OK, it's a ranking system, the better students will probably solve the questions more quickly and will have a chance of answering all the questions, but I still think exams are supposed to test knowledge and not writing agility.
3. This one was an exam that really tested knowledge but we had only a small set of sample questions (without solutions) to study by.

Well, after telling everything I hate about studying I can now rest and don't think about it for the rest of the summer.

Education, Part I: Masochism

2008-07-10T22:29:00.003+03:00

Know this thing when bugs rush towards the light that's gonna fry them? Well, I sometimes feel like one, except I have more than a single chance.

What am I talking about? Last week I've finished the last test for the semester in my Master's studies and I can't help but feeling a great relief and thinking to myself "pheeeeew, done at last, three months of freedom, here I come!". And then I recall how it all began...

Before starting my Bachelor's degree I was excited - "WOW, University, that's going to be interesting!" (a bit geeky I know, but no hiding the truth here) - I think it took something like a couple of weeks to get me totally frustrated and counting the weeks 'till the end of the semester.
Before the 2nd year I thought - "Done with (almost)all the math so it's definitely going to be better this year" - well, not really(maybe a bit), but at least I started getting higher grades.
Before the 3rd and last year it was something like - "Well, I'm done with most mandatory courses so now I choose most of the courses I take, how bad can it get?" - and indeed, it was better but at the end of the year and the degree the only thing I could think regarding further education was "Nice it was, but a B.Sc. is more than enough for me. Thanks a lot dear university but I'll probably leave the Master's to someone else". I was truly sure there's no way I'll go for a Master's, at least not in the near five or so years.

Well, it took me one year to start considering the issue and another year to actually start studying, and again same old lies "It's a Master's so the courses are more interesting and less demanding", well it has some truth to it, but no matter what, at the end of the year I'm still exhausted and happy to get rid of all the courses I was so excited about taking.

So now I have three months to forget all I've written about here and get re-excited, you can count on me.

Talking about masochism, this week I've learned it can take the form of curiosity, I won't tell you what I'm talking about - this will be sadistic - so as translated directly from Hebrew "who understands, understands".

Light at the End of the Tunnel, Part II: The End?

2008-07-03T22:30:00.003+03:00

Well, it's time I conclude this whole upgrade story.

The other major problem I had, still have actually, is related to memory consumption. Apparently 10g processes consume more PGA, I haven't really understood yet if that's a planned behavior or a bug, but the fact is that just opening a forms' session process now takes something like 8MB of PGA instead of 3MB on 9i add to that the fact that I have more than a couple concurrent users that do more than just opening forms (they actually work with it), throw in my operating system (Windows 32bit), add to the mixture memory leaks in forms (mainly in the inventory module) and you might as well call Huston.

I had some advices from Oracle on this one but no real solution. I've tried reducing the stack size for processes using orastack, an Oracle expert (same one from here) suggested I ask the heavier users to close forms they don't currently use and found a yet unsolved bug that seems relevant, and I still have an open TAR regarding the issue.

So where do I stand now? I'm not really sure. I used to have the database crashing with ORA-7445 and ORA-04030 daily, and then it just stopped - yeah, I do face an occasional crash once in a week or two but somehow the problem is not such a pain in the ass anymore. One of the possible causes is a reduced number of concurrent users, it seems that it's just enough to save me from the fatal memory consumption limit.

I'll probably let you know if I have any insights and I do have a lot to say about the increased memory consumption of 10g but that's stuff for a post with a different title.

Just Stupid

2008-06-28T23:10:00.007+03:00

Well, I was actually going to write the second (and probably final) part of Light at the End of the Tunnel but some things are just too provoking (and I'm not at the end of the tunnel yet anyway).

My folks bought a new phone device today, probably the cheapest device in the whole store, but that's not really a justification for the piece of crap this device turned out to be.
What can be so terrible about a simple phone device? Well, I'll supply an example, and please keep in mind that this is after only a ten minutes interaction with the device, I'm actually a bit curious to discover what other flaws the device has...
To the example then. My dad opens the box, connects the phone and tries to dial... nothing. Apparently someone had a "brilliant" idea, the device has a lock (see picture below) that somehow reminds the ages when computers used to had locks - didn't find an image of that one (and didn't look to hard either), anyway, the purpose of the lock is to block calls to certain numbers: it has three modes - dial any number, don't dial any number and dial only numbers that don't start with an '0'. Those three modes are represented by three colors: red, yellow and green, at least it's trivial which color represents which mode, that's right, red is "dial anywhere" and green is "forget about dialing", someone else thinks that's extremely stupid? If you take a close look at the picture you'll also notice that there's no real way to know at what position the lock is now (yeah, it's not green), very amusing. What adds to the amusement, considering the fact that the phone didn't came "red" by default, is that the Hebrew version of the user's guide doesn't have a single word about the lock.
And if that wasn't enough I have another example. The device has a speaker, so you'd probably expect to be able to dial a number, press the speaker button and have a conversation rolling, not with this one. Once you press the speaker button the number typed disappears and you're welcome to type in a new number, so you have to press the speaker button before dialing or use some commando techniques such as pressing the redial button after the number you typed has dissolved - that's a bit weird since you didn't really "dial" that number before, oh well.

A home phone device is supposed to be as simple as possible to use and trivial to understand for everyone, this one has definitely failed this test. Just Stupid.

Alerts

2008-06-24T22:38:00.003+03:00

Last weekend I woke in the middle of the night to discover my arm totally numb, and when I say "totally" I really mean it - I couldn't feel it nor move it.
I don't know if I'm some kind of freak or maybe it happens to everyone occasionally but it's definitely not the first time it has happened to me. What I usually do when it happens is to grab my "dead" arm with my other arm and throw it off the side of the bed - I do it to hasten the blood flow back into my arm (have no idea if this actually has any effect though) - this is actually the creepiest part of this whole process, because when I grab my numb arm with my not-numb arm I don't feel like touching myself and the arm is simply dangling like a dead piece of meat until blood flow is restored.
Up until now the arm revived each time, but I always wonder to myself what would've happened have I not woken up. I also wonder if what got me awake is some alerting (hence the title) mechanism of my body, and since I don't tend to wake up in the middle of the night this is probably the case.

What this short tale got to do with system administration (except happening to a sysadmin), well, nothing really. But it did got me thinking that I should write a post about the different monitoring and administration tools I use to detect and analyze bad things in my system, of course, I still care more about my own arm - can't type commands too fast with just one arm :).

Information Boom

2008-06-13T00:14:00.002+03:00

I'll take another break from my upgrade aftermath tales to write about an issue I think many sysadmins face, and not only sysadmins. Handling information.

It seems that since I started this job I'm in a constant race to beat the accumulation of information, and I never seem to be on top.
When I started my training I had a notebook in which I wrote all my remarks, since at the beginning you write any minor note (although later it's trivial anyway) I soon started a second notebook. But notebooks aren't so efficient so after starting the job for real I've started to use OneNote, WOW! I can create plenty of pages in a single tab, this will definitely do the trick! But soon I needed more tabs and then more OneNote notebooks too.
To store documents my team (a team of sysadmins) uses a SharePoint portal, we used to have a folder-hierarchy structure to our documents, but at some point it just got too messy so we've decided just to throw everything in the root folder (chaos indeed) and tag it, then of course we've created views and filters for the different tags.
We also have a Wiki to store and share important knowledge.

But even with all those mechanisms set in place it seems there's just too much information to handle:
I have my daily tasks to manage. Every new topic (regardless how minor) I deal with is a page or a tab in my OneNote. A major upgrade or installation consist of my own notes (what I did, what errors I got), official installation/upgrade notes, know bugs documents, new features' documents etc.. Every problem has a related metalink note or my own document describing how to fix it. And except the EBS I'm also in charge of an OID and an iAS servers.
That's just a partial list and nothing in it can be skipped.
Both external(e.g. supplied by the vendor) and internal (written by the sysadmin) documentation is critical to every sysadmin - you can't memorize every parameter and functionality in your system and you can't expect to remember every error you handled three years ago.
But the problem starts when you need to retrieve this information, if you don't organize your information wisely it doesn't matter that you have a perfect documentation of how to fix a bug because you just won't be able to find it, too bad. I can think of many times when I was sure I saw something familiar before and had to start guessing where exactly - my OneNote? The portal? Maybe in my documents' folder before I started using the portal? Maybe I eventually didn't download it from the metalink? Or maybe someone else showed it to me? Trust me, this can get pretty frustrating.
Having a good logic for storing your information is crucial but from my experience it's just not enough as information tends to outgrow any such solution.
As I see it, the only real solution is a powerful searching solution that has access to all the storage solutions you use.

I'm actually going to explore this field of enterprise search solutions so I might have more insight in following posts.

Light at the End of the Tunnel, Part I

2008-06-06T17:52:00.004+03:00

This post should actually be titled "Upgrading Part IV", but my Roman math is not so good - I admit checking if I should write IV or VI - so I decided to stick to the simple numbers.

After bitching about it a lot, it's time I tell you how my inconsistent problem got solved.
This one was too much for me to solve on my own so I had an Oracle expert come over. After describing the issue I'm facing he said we should set up a trace on all client connections from the 8.0.6 home - we had to do this for all connections since we had no idea which concurrent will get stuck. It took some time before we realized that we had too small a pool of trace files and that trace files we were looking at didn't belong to a stuck concurrent anymore. After doing some re-configuration on the trace settings we finally found a trace file that belonged to a stuck concurrent.
Boy! The guy has patience. As someone on my team said, a good expert can be measured by the patience he demonstrates going over log and trace files.
Going over the entire trace file we discovered that at some point during connection establishment the process just stops, we could see that the client was waiting for some input and... nothing. Although it felt like closing on on the problem it seemed like a long and tedious process will be needed through a TAR, since however good the expert I had on site is, it's just not his specialty, and it definitely looked like some internal bug.
Before leaving, the expert noticed I had NTS defined in my sqlnet.ora file for SQLNET.AUTHENTICATION_SERVICES, I explained I was using it to logon as sysdba without entering a password. He suggested I try to remove it, so I did.
At this point we were towards the end of the day so I didn't expect any concurrents to get stuck anyway. But the next day the problem disappeared, I was thrilled, seems like the NTS setting was the problem all the way! Thinking about it, there's no chance I would've thought of it to be the root of all evil. I think the only reason the expert noticed it is because he's not used to dealing with installations on Windows, it seems to me that even he didn't believe it will solve anything and suggested it just as a shot in the dark.
Proper disclosure. The expert suggested some other changes as well. Those were changes to DB parameters as specified in an Oracle document regarding DB parameters for an EBS installations.
The changes had effect only on the next day (the day everything worked) as well, so I might be wrong and what really fixed the problem was altering those parameters. Since I'm curious and somewhat suicidal I'll probably have the NTS returned and check if the problem reoccurs, promise to let know if my intuition here is wrong.

The funny thing is that just after discovering this issue with NTS, I found another one, working on an entirely different thing. Try googling for "NTS sqlnet.ora".

Well, one problem down, one more to go. Probably will tell about it in my next post, although it's still not solved.

Widening My Horizons

2008-05-24T17:57:00.002+03:00

Well, I've decided to take a break from my series of upgrade related posts and write about something more fun.

This week I got a new 22'' (wide) monitor as a replacement for my old 17'' monitor. No doubt this is a great improvement, so many things look different now:
Most emails fit the screen without scrolling, I can accommodate more tabs in my Total Commander, I can comfortably rearrange my OneNote pages - it actually feels like each page is twice as wide, I can open two Terminal Services side by side and work with them in parallel, it's easier to read documents - less scrolling, and those are just the things I could think of on the spot after one week's work. I'm sure I'll discover many more advantages. Oh, the joy of having a new toy!

Of course, there's a downside as well(not that I really care, but...), I'm going to get spoiled.
I think it's relevant to many aspects in life, when you get a taste of a better something, you can't really go back to what you previously did just fine with - and yes, I was pretty satisfied with my 17'', although getting an occasional "man, how can you work with it?!" from guys with bigger monitors. And now I'm going to become such a guy.
It reminds of when I was a kid and played Warcraft (the 1st one of course). I played it on my computer and was just fine with it, but then I was at a friend's house and he had a faster PC, after playing a few levels on his PC mine seemed so sloooow, it was impossible to play.

My conclusion is that such improvements are good only if you know you won't have to revert back to the previous situation, otherwise it's going to take a long time getting used to and might even be frustrating.
I guess I'm going to demand a 22'' monitor to be included in every contract I sign from now on...

Upgrading Part III: Consistency

2008-05-10T23:46:00.004+03:00

Well, I'm still battling problems after the database upgrade.
I think that the most annoying thing about the problems I'm facing is that they're not really consistent.

Of course, we'd all like a problem free world, but once we encounter a problem the first thing to do is to reproduce it, thus assuring ourselves we know what we're dealing with.
First you try to reproduce it on a development environment if that doesn't succeed, you at least expect some consistency on the production environment.
When you don't have this either you're in a trouble, you might do something one day and hope to see results on the next day and indeed the next day problem still occurs but in much lower rates and you think to yourself "I must be in the right direction" but then the next day business is as usual - the system is in a total mess. Then you understand you don't understand anything. The next step is to try and reproduce problem on your terms.
In my case the issue is with concurrents randomly getting stuck. So I first tried to see if the problem is with all executables or is EBS related, starting with simple cmd scripts running non-stop and going to submitting multiple concurrents using CONCSUB - at last one of my manually submitted concurrents hangs. But I got no real insights from that as well. No consistency whatsoever.
One of the main issues with inconsistent problems is that they're hard to describe - in a TAR for example - most directions for detecting the problem involve some major configuration problem, one that would prevent the system from working altogether, but remember, I don't have this kind of blessed consistency.
Another thing with inconsistent issues is that you come to work and hope things will get bad so you could poke the issue around a bit more and so you won't get that annoying voodoo feeling. Usually a sysadmin's hope is that nothing will get bad...

Actually, this week I had some progress, so hopefully the next post will contain a bit of advice that can potentially save a serious headache to someone if he stumbles upon this blog while trying to resolve a similar issue.

Upgrading Part II: Bad Job

2008-05-03T18:32:00.003+03:00

As with any upgrade/major installation I've ever made (at least as far as I can remember), after upgrading the database to 10g the application have demonstrated some interesting errors. One of the problems is related to concurrents and the executables they spawn crashing in mid-air - promise to tell about this in more detail when I myself have any idea.
At some point I suspected that the problem was with custom code running cmd scripts. Since some of those scripts are called from within a PL/SQL code using a Java stored procedure I thought that maybe I should try and use a "less custom" way to do that.

Luckily (or unluckily), I just read the previous week about a new feature in 10g - the dbms_scheduler package that's supposed to replace dbms_job and to be much more powerful, for instance it enables you to run cmd scripts. So I thought I'll try this out since it sounds exactly like the built-in method I was looking for.
Well, I have only one thing I can say about this: it's better to leave a feature out of the release than to keep it in when it sucks, totally.

Really, my keyboard is still soaked with sweat from my efforts to run a single script that echoes some text.
I've already written about this but I guess the message didn't get through, if I'm supposed to start the Oracle Scheduler service to run jobs then that's exactly what I expect to be written in the error message I get when not doing so - certainly not a "file not found error". I also expect this to be written in any (well, at least some) documentation describing the new feature, that's not the kind of things one supposed to dig up only in forums. By the way, some hilarious problem with similar symptoms I read about while searching for a solution to my own issue - I've actually tried this out since at first I thought this was the problem I was experiencing - it appears that for some users (maybe in earlier 10g versions) just supplying a cmd script didn't work, they had to run cmd.exe with parameters. Can't even begin to understand how you manage to create this bug and release it.
Well, after completing the POC (if at a the cost of health) I got to the real thing running a script with parameters. In some document I saw something like the following example:

dbms_scheduler.create_job(job_name => 'JOB', job_type => 'EXECUTABLE',
job_action => 'script.cmd', number_of_arguments => n);
dbms_scheduler.set_job_argument_value(job_name => 'JOB',
argument_position => 1, argument_value => '...');
...
dbms_scheduler.set_job_argument_value(job_name => 'JOB',
argument_position => n, argument_value => '...');
dbms_scheduler.enable (name => 'JOB');
dbms_scheduler.run_job(job_name => 'JOB');

Well, maybe it's just my system that is a freak but apparently the enable procedure erases the job. Exactly, I create a job, I run dbms_scheduler.enable and no job in the table - no error message, no nothing, I might have run drop_job instead. Well, apparently I don't need that line anyway. After some more struggle with cryptic error messages I got the package to do what I wanted it to do, but that's really not good enough.
I can't even start to imagine what kind of efforts are needed to bootstrap all the advanced scheduling features - windows, chains etc..

I'm willing to bet money on the fact that most developers would have given up much earlier than I did saying this stuff just doesn't work. You can develop useful and cool new features all you like but if you can't cut/paste an example and see it just working no one would use it, if all users get when something is wrong are misleading error messages they'll just get frustrated (and you can see I am one such user) .
I first read about this feature in a "10g Top 20 New Features" document, and indeed it sounded great but if it's impossible to use it, it's not really a new feature at all.
I'm sometimes not sure if I should account all those funny errors I deal with to the fact that my system is on MS Windows - a less common platform, maybe I should. But that's not a good enough reason, if I'm in possession of a disk labeled 10.2.0.3 for Windows I expect it to work, I don't really mind having it released half a year later than the corresponding Linux version, I just want it to function.

As for me, I know about this feature, I can even make it work, but there's no chance I'll suggest it as a solution to any need unless as a last resort. Too bad.

Upgrading Part I: Human Error

2008-04-24T12:53:00.005+03:00

In my previous post I've talked about a problem I experienced before upgrading my EBS database to 10g, well, the upgrade has been performed and I think it has supplied me with ideas for at least three posts. Of course the bug experienced on the test environment didn't show itself on production (I didn't make the fix intentionally to see if it re-occurs), and what I have now is a very "nice" variety of other bugs. But why rush to the aftermath? Lets start with the upgrade process itself.

I had all the upgrade procedure in soft and hard copies, have made about 4 upgrades on dev and test environments, was pretty sure I would successfully handle any problem that will pop up. Actually, many steps of the upgrade process came down to a Copy-Paste action from my upgrade document to the command-line. It's so frustrating that what eventually got me delayed for about five(!) hours is a totally stupid mistake on my behalf.

Of course, before such an upgrade a backup should be taken, I have a script that runs every night: it brings down the system, a backup is taken (we use EMC's BCV) and then the system is brought up, all I had to do is re-schedule this script. Being a smart ass I've decided to save some time and disable the startup script, I only need the database up for the upgrade process. I could think of plenty ways to do this but decided that emptying the startup script will be the most secure - this way I'm not touching the main backup script. So I've made a copy of the startup script and... emptied the shutdown script. Yes you read it right, no typo.
So the time comes, I'm watching the backup logs and WOW! that was fast, usually it takes about five minutes for the shutdown process to complete and now the system is up after one minute, great! Hmmm... maybe not so great after all. So I take a closer look at the logs and discover what I have just done, the meaning is that the BCV split was performed on an open database and it's worth nothing (well, maybe not nothing but that's a bad idea to start the upgrade with this kind of backup). Oh, I'm going to use some EMC terms but it's OK since many of them I myself discovered only during this upgrade.
Well, actually there's no reason for panic, the backup devices are mostly synchronized and all I have to do is shutdown the database and take the backup again. I've called the guy in charge of storage and asked him if I'm right, he said I'm right and guided me through the process of taking the backup from the command line. But, alas, That's not so simple!
Apparently when EMC tells you that a split is complete, well, it's lying. The split process (again, if you have no idea what I'm talking about, just imagine some backup process) continues in the background and until it's completed you can't take another backup. The more frustrating thing is that the storage guy didn't know about this (he's new, cut him some slack) and I didn't know how to check the progress of this background split. At some point - when only one disk was still splitting and the others were already re-synchronizing - I've discovered how to check the progress of the split. Great, I'm 99%, but boy this 1% took it's share of time to complete.
Phew! At least now I know where I'm standing. Well not exactly, EMC has another "great" feature, when it comes to synchronizing the last parts of disks it take a totally disproportional time (remember the 1%), so when there were only 50MB left to synchronize the machine had to work forever for every 0.1MB, no kidding. So I didn't really have any estimation as to when I'll finally start the upgrade process and I was already about 3.5 hours late, and this is a downtime we're talking about here, we usually like to minimize those.
So I waited. I guess this is what they call "paying for your mistakes".

Eventually the synchronization completed and I got to the upgrade process. The upgrade was perfect, no unexpected errors and the entire process was completed (long) before scheduled time - thank God for buffers.

Well, a nice prelude indeed...

The Persistent Bug

2008-04-09T22:43:00.004+03:00

My system has a bug that has been around long before I became the sysadmin, it reincarnates each time in a slightly different form, but for me it's the same old bug.
I call it "the -107 bug" because it's main phenomena is an error message in the concurrent log that specifies a long number - -107... (I don't really remember all the digits, there are lots of them), it looks to me like a really big integer going over to the negative side or something like this, but I don't really know.

I was first introduced to this bug after installing some major patch (maybe ATG_PF.H) when QA alerts started failing in every execution, then, other (with no pattern I could pinpoint) concurrents started to fail with this error as well - but not all the time.
I've opened two separate SRs to handle those issues and over time some actions have been made to improve the situation and eventually a patch that was supposed to fix the problem for the concurrents had pretty much fixed the problem for the QA alerts. In the last half year or so those errors became really rare.
But then, yesterday it hit again. During the testing session before upgrading the production to a 10g database, QA alerts started to fail with this error (although sometimes doing their job), eventually the errors have stabilized themselves to target only specific transactions. Getting desperate I came to the point of telling my boss I believe the best action plan is to perform the upgrade anyway, I had several arguments:
1. This bug might skip the production.
2. We have a project that will alter the QA methodology to a new one in which this specific action won't be performed.
3. I know this long enough to know that it won't be solved in a reasonable time frame.
4. Schedule considerations.

But something was different this time, the pattern was too consistent, usually this bug is unpredictable and actually never was reproduced on a dev environment. Indeed, a few minutes after my conversation with the boss, one of the team members found the problem - total voodoo crap if you ask me.
The INSERT action this specific alert performs had a section that read "-1*&some_parameter" changing it to "-&some_parameter" solved the issue.
OK, let's say the new and shiny 10g doesn't know how to digest a -1*x expression, but the alert won't fail when it's the only one triggered as part of a transaction (I don't really know the technicalities of this whole process so I hope my explanations are clear enough), it only fails when other alerts are triggered as well.

Anyway, I guess that's another score for me in the endless struggle against this stubborn bug. On the other hand I'm pretty embarrassed, I did tell my boss there's no way to solve the issue...

Wake Up Call

2008-03-29T22:26:00.003+03:00

Growing up is the process of discovering that all the great stuff you believe in are just fictions and illusions. Santa isn't real, your daddy doesn't really know everything, guys who write Tetris(Tris) for the iPhone don't really block the possibility of moving the pieces upwards and real-world software isn't really developed and tested like it's described here.

Actually, I've just experienced an example of what I'm talking about in those few first lines of this post. How do you place a link in a post? Very simple, you mark some text, push the link button and enter the desired URL. But then, everything you type after the link is considered part of the link. My way of overcoming this obstacle is to edit the HTML code directly to get outside the link tag, I'll really appreciate simpler solutions. Now, I AM using Blogger and it's pretty good but this little bug is just pathetic.
When I was younger I really thought that these major software products (except Microsoft that always was a subject for mockery) are written and tested to the last bit and that they really do work - otherwise, why would someone use and even pay for them?
But when dealing with the internals of software like a sysadmin frequently does, you sometimes discover code sections that obviously weren't tested and maybe even not thought about for a single second. Just some of the examples I've encountered: code that exits instead of returning an error code, irrelevant localization code that slows down performance tremendously, code with zero error handling and much more.

It's a little sad, like every myth you discover is only a myth.
But in second thought, that's why guys like me have a job.

New Technology

2008-03-20T23:22:00.002+02:00

Usually my tasks as a sysadmin are pretty standard: upgrade the database, fix the bug, deal with the backup issues and so on. While I might have not upgraded any production database yet, I still pretty much know what I'm dealing with.
But every once in a while I'm required to deal with something (almost) totally new. For the last couple of weeks my team leader and me have been working on implementing EUS - or more simply: database user authentication via OID using Kerberos. While we both pretty much know our OID, we haven't implemented this specific solution before (at least not one that worked :-)).
I really like those new challenges, I learn something new each day, but this kind of stuff makes you feel like you can add a new bullet in your CV. Like my team leader mentioned, it makes you want to work only on this new and shiny "toy" until you finally get it up and running and forget about the more mundane tasks. Unfortunately, those mundane tasks are often more pressing and so we end up working between other tasks and making little (if in a steady pace) progress each time. Well, life's tough. At least we ARE making progress...

Overtraining

2008-03-15T20:10:00.003+02:00

Metalink is one of the most useful tools for an Apps' DBA, you can search it for articles or upgrade notes, download patches and search for bug resolution instructions.
The first time you search an error message and get many results, you go over nearly every note, you don't really know to distinguish the relevant from the irrelevant and since you're probably new to the job you're afraid to miss the desired solution. Over time you develop a sense for what is relevant, even if the note doesn't mention the exact error message, you sometimes know that the issue is the same. On the other hand, just like with machine learning, you might get a bit too confident in your ability to distinguish relevant from irrelevant and miss valuable notes.

That's what happened to me this week. I was looking for a solution to a problem with an engineering form, I found a note that mentioned a patch that was supposed to install a new version of the form, but applying the patch didn't upgrade the form. So after doing a bit more searching I've asked from one of the analysts to log an SR (I wouldn't really know how to describe an engineering issue). Luckily, he was wise enough to search Metalink himself first, and indeed he've stumbled upon a similar issue in which it was suggested to make a full installation of the engineering application. Now that I think about it, it's trivial and probably the first thing I should've thought about - a form might not get installed if the application is only in shared installation mode.
Anyway, this fresh perspective have saved logging an SR and probably a couple of days work.

Less is More

2008-03-08T18:35:00.002+02:00

This week I had to bring back to life an environment I'd kept "frozen" for more than a year because of a revived project, kinda like Demolition Man. I've cleared a server to host this environment and thought I had a creative idea:
Nowadays I use NetApp storage (through iScsi) for this server, mainly because of the snapshot option, but this server has also a local disk that can accommodate the database. So I've decided to use the local disk - I get to keep the environment I had on this server before, the robocopy is much faster and I won't need snapshots for this environment anyway since it's for reference purposes only, not for development.

Do you know this guy Murphy?
A few days after the environment was up I've discovered that a couple of ApEx applications that were developed on this environment weren't kept anywhere else, so I had to export them and then import to the environment where the rest of project will be developed. Yes, I know, someone has real bad development practices, but, as my boss told me, part of my job is to clean the mess others make. So I've tried to open ApEx - nothing, a blank page. Now, you see, when those ApEx applications were developed ApEx was relatively new, actually, this old environment had an ApEx 2.0 (when it was still referred to as HTMLDB) schema as well as an ApEx 2.2 schema, and the only version to ever reach production was 2.2, so at first I thought there might be some misconfiguration of the iAS server or something like this. After exhausting the few ideas I had regarding misconfiguration I've tried to export the applications using APIs and got a strange error message about a missing datafile.
Well, you learn new things each day, it turns out that even if you can see the tablespace's definition and see the package specs for packages in it, it doesn't necessarily mean that the datafile for the tablespace is present.
It turns out that when I recreated the control file for the database I didn't have the datafile for the ApEx tablespace in the template script (remember it was new at the time). I'd expect the database to shout at me something like "man! you're missing a datafile" instead of just generating some nonexistent path for the missing datafile and going around acting like everything's ok.
God I wish I'd used the NetApp option.

I had two options:
1. Try to re-integrate the additional datafile into the database.
2. Restore the database again.
Now I get to the meaning of the title of this post. Re-integrating the datafile, if successful, will save precious copy time and giving up on a challenge is against my nature (and against this of another DBA from my team who tried to help me). But, during copy time I can do other things and this re-integration thing didn't look like it has high odds, and so we would probably end up with the second option anyway.
So, what I'm trying to say here is that being a good sysadmin doesn't always mean you have to solve any single problem, sometimes you just have to pick the more efficient course of action even if it is the more trivial one, even if it means you leave a problem unsolved. In life you have to pick your fights wisely.

My Good Deed

2008-02-26T20:52:00.003+02:00

Those of you who have keen eyes have noticed that I've added a new link in my favorite blogs section. I've stumbled upon Aviad's blog this week when I've encountered a problem while upgrading my EBS database to 10g. Actually that's the second time I encounter this blog, the first time was when Steven Chan mentioned it in his "Notable Apps Blogs" list. This first time I was just proud to see an Israeli in Steven's list and didn't linger too much in the blog (although it did strike me as quite an impressive blog).
The second time, even after I've discovered the specific problem I had on my hands wasn't mentioned in the blog (although many others were), I did some more thorough browsing. The guy really does excellent job, his blog might have saved me a lot of time if I was aware of it before.
Anyway, this blog inspired me to stop complaining for at least one post and share some useful information, the issue I had looks like a good place to start.

As mentioned, I'm in the process of upgrading my EBS(11i) database from a 9.2.0.5 database to a 10.2.0.3 database. During the upgrade phase (after the database software is already installed) I've encountered an "ORA-12631: Username retrieval failed" error message. The problem, apparently, was that I didn't have a listener up and running. Having a listener up might seem very basic to some of you, but I was working with my own upgrade notes that were compiled based two successful upgrades. More than that, DBUA starts the database on its own and I'm pretty sure it did the same thing with the listener on my previous attempts. I don't really know what was wrong this time but I did get back to the original notes by Oracle to see if I might have misinterpreted something.
There's definitely some confusion here: Note 362203.1 (EBS upgrade note) orders the stopping of any existent listener while the database upgrade document (not EBS related) mentions the need to create a new listener using NetCA if I don't already have a listener. So, does the upgrade manual mean I should have some listener up? And if so, why does the EBS note tells me to stop all listeners?
Again, if the stopped listener was indeed the issue, I'm sure there was some more sophisticated way to phrase the error message.

Well, had some bitterness in my post after all.
At least this might be useful to someone.

Power To The Numbers

2008-02-16T15:46:00.002+02:00

Our IT department (which I'm part of) has an internal Sharepoint based Wiki, our aspiration is to make it, as Wikipedia is, the place you start from when looking for information. It's supposed to include all kinds of information: general technological issues like how to install a VM, more specific issues like how to change the infrastructure for an IAS and even not work related topics like take aways to stay away from.
We're not at all there yet, and thinking about it I'm not sure we'll ever really get there. I can think of a several reasons:

1. A Wiki page is not really that good a replacement for a real document, a Wiki page can contain summarized information and maybe a link to the original document, but unless you have trouble finding the document, you don't really have a reason to look in the Wiki in the first place.
When looking for some arbitrary subject on the internet it's not always trivial to find a good document about it, so you use Wikipedia. In an IT department when looking for a documented issue it's simpler to just look for the document, since whoever documented the issue have probably written a document even if he wrote about it in the Wiki as well, and as mentioned, documents are better.

2. Specific issues like the mentioned example are too specific since they often address only one person, the one who wrote them down. Sure, people can forget and get replaced but still a document will do the job just as well.

3. The main reason as I see it.
A Wiki exemplifies the egg or chicken paradox. For people to use Wiki you obviously have to have some content, but for people to add content they usually have to consider the Wiki a source of information people use, not much point in writing something if nobody reads it.
Well, Wikipedia DOES work.
But let's say you have a blank new Wikipedia. Right, according to my assumption no one will
write anything, but after all, IT IS the internet. Even if one of every one thousand users writes something you get plenty of data, so now a higher percentile of the population is willing to donate their knowledge and so you get a plague-like effect where people use Wikipedia, find what they were looking for and add their own insights.
Now, lets get back to our department of about 30 people. Sure, we did write some pages at the beginning and even had a "Wiki week" in which we tried to add as much new content as possible, but as long as it's easier to ask the guy over the table or just browse for the right document, the Wiki usage seems pretty artificial. I've written some pages in our Wiki but I can't think of a situation where I'll actually think about looking there for answers.
That's why I'm trying to make the trainee to document everything he learns in the Wiki, this actually achieves some goals, first, as a newbie he has a better sense for what should be documented, second that's a good way to create a sort of a "training Wiki" and third it makes him go over what he have learned - everybody's in gain.

On the optimistic side, maybe it'll just take time, maybe in several years our Wiki WILL be considered a central source for information.
We'll just have to wait and see.

Syntax Confusion

2008-02-08T00:14:00.000+02:00

There are many programming languages, each has it's strengths and weaknesses. And it's own syntax. I don't really understand what is it good for. Sure, each language has it's unique features, a strongly-typed language will have different syntax from a weakly-typed language, but I'm not talking about this "advanced" stuff, I'm talking about the very basics. And no better way to show what I mean than an example:
You already know I'm not a programmer, but I write PL/SQL code for the application, cmd and Perl scripts for administration purposes and this last semester I took Computer Vision and Image Processing courses and had a lot of assignments to implement in Matlab. Four languages, four different ways to write a comment. At some point I got confused with all the '%','#','--' tags (REM stands out more clearly), is this really necessary? Why should there be three(!) different ways to write a simple "else if" statement? Ok, let's say "elseif" somehow simplifies parsing, but "elsif"?! That's really ridiculous, it just looks like someone was trying to prove his creativity.

Why won't some standardization organization decide on how the basic syntax (comments, loops, if-then-else statements) should look like, and make - I don't really know how those things work - the vendors to have the next versions of their compilers to support the standard (unfortunately, backward compatibility is inevitable here)?

I don't really know if my idea is in any way feasible but this syntax confusion can get very... well, confusing.

Life According to Pinball

2008-02-01T00:25:00.000+02:00

Since my days as a B.Sc. student I have the habit of playing pinball while doing assignments or studying for a test, no, I'm not saying I'm solving questions with one hand and hitting the ball with the other.
It's just that after solving a hard question or completing a test I have the urge to somehow "reward" myself, having a snack works as well but it's a bit more fattening. So when I want some short break (after all I have more questions to answer) I turn to pinball(Windows pinball). Well, I admit playing spider solitaire for the same purpose but pinball has a big advantage: after playing for a while it's possible to play mostly with your instincts letting your mind wonder around and sometimes creating a jogging-like effect, that's actually how I got the idea for this bizarre post.
I also think that pinball can be pretty educational. I'm not kidding (well maybe I'm not THAT serious).
Here are some lessons that I think can be taught playing pinball:

1. Think well about your options before leaping to the extreme solution.
Imagine the next situation: the ball is descending quickly and it seems like it's going for the middle (and your Center Post is not on), the reflex of most people is just to hit hard and fast with both flippers, WRONG! (at least most of the times). From my experience the best options are either to stay cool and it might be that the ball WILL hit one of the flippers or hitting only with one flipper hopefully diverting the ball onto the second flipper (remember, if you hit with both flippers the second flipper isn't there to catch the ball).

2. Timing is everything.
This one is pretty trivial in pinball, it's not enough to do the right thing, sometimes you must have perfect timing. It's not enough to hit the ball, if you're after the Reflex Award (and maybe completing the launching mission) you have to be very precise.

3. Live the now.
It doesn't matter if you've messed up the first two balls, you might still break a top score with you're third ball. And vice versa, scoring 10 mil. with your first ball doesn't guarantee you anything about your next two balls.

Looking At The Mirror (2)

2008-01-27T23:21:00.000+02:00

About two weeks ago I've started training a new Apps DBA (my replacement actually). However exciting, this mission (as I was warned) is very time consuming, it's not only the time spent on explaining what does a 'system' mean and the more specific niches of the system he will have to manage, but I also, as the new guy makes progress, have to show him how to deal with day to day issues, how to apply a document to the system and how to deal with annoying developers.

Again, as in the first part of Looking At The Mirror, I find myself recalling my own training a couple of years ago and, as reminded by a friend, I remember being pretty sure there's no way I'll be able to really do something productive like my predecessor did, I thought I might be able to bring the system up at most. Well, as it is with those things, I was able to do a bit more than that. One day you just realize you're able to handle the system on your own and even more than just 'handle'.
By the way, talking about bringing up systems, today the new guy brought up two development environments on his own, I had to explain some things I've skipped before as he went along but eventually the environments were alive and kicking.

And that's another thing worth mentioning. When you know what you know, everything looks trivial. It's only when you have to transfer your knowledge to someone else that you realize how much experience you've accumulated and how tricky this all system thing is.
The bottom line is that there's really no way of transferring all you know, you can pass part of this in documents you've written, you can give oral explanations and tips, but eventually experience is the best teacher and being thrown into the deep cold water of a system crash(which will probably happen the next time I'm on vocation( - Voodoo part)) is sometimes better than weeks of explanations.

I'm pretty curious to see how the new guy handles the real stuff. For instance, if everything goes on as planned, this week he will watch while I upgrade an EBS database to 10g and in the next upgrading phase I expect him to do it on his own (well, maybe with SOME help).
Things are definitely going to get interesting... for both of us.

Looking At The Mirror (1)

2008-01-18T13:27:00.000+02:00

This week I've been interviewing candidates for a programmer's position.
Up until now I've always sat on the other side of the table, so it felt strange to be the one asking the questions and not the one trying to impress by answering them.

This is not a simple task.
When you grade a candidate at the end of the interview you(or at least I) not only consider his ability to answer your questions in a satisfactory way, but, inevitably, you also think whether you would grade yourself high given the considerations you've applied, I think this shouldn't be a factor and neither the fact that you tend to sympathize more with people who have a similar background to your own background. Well, we're all just human.
Actually the considerations I've applied in many of the interviews are even more "unprofessional":
The interview consists of two parts, in the first, an HR representative asks personality questions and in the second the professional interviewer(me) asks professional questions. But actually , by the time we get to the professional part you've already formed quite a firm opinion of the candidate since virtues like the ability to operate as part of a team or the ability to quickly grasp new technologies are all part of being a good programmer(and most other technological positions as well). So sometimes the professional questions are asked only in order to validate what you've already realized about a candidate.
You can probably say it's not fair to decide on a person's future (a bit dramatic but true sometimes nonetheless) based on a first impression, that's true, but life's tough and in reality our interactions with other people are often based merely on first impressions. Besides, the interview has only limited time so first impression is mostly what you have.

Time To Let Go

2008-01-10T23:49:00.000+02:00

In job interviews you sometimes get asked about your bad qualities, among other things I usually mention me being a "too hard trier". It might seem like me trying to be a smart ass and slip a good quality as a bad one, but I really do think I sometimes try too hard.

I bet everyone knows this thing when you sit for hours trying to solve a problem just to get frustrated, go home and wake up with the now trivial solution the next morning. And that's exactly what I'm talking about, of course it's great to come up with a solution just when you're about to give up, but isn't it even better to skip the frustration part?
I don't really know the exact neurological explanation but a good night's sleep or even setting your mind to something else definitely triggers some processes in this head of ours that help us see the problem in a different light, and that's always a good thing. It's like when you try to remember the name of a song/movie/book and can't pinpoint it no matter how hard you try, but it will always pop into your mind later without the slightest effort.
This approach actually goes along well with what I've written in a previous post about counting on luck but somehow when it comes to letting go and letting nature do it's magic I seem to stay stubborn.

I'll try to continue my self-education and my advice to everyone who has a similar issue is to do the same, sometimes it just doesn't worth the trouble (but sometimes it does...).

Just Tell Me What Did I Do Wrong

2008-01-04T19:36:00.001+02:00

Know that thing when your girlfriend(or wife, or whatever) starts acting like a walking icicle because you've done something to upset her? So you also know that she will put you to the test of discovering your error yourself. No matter what exactly you did you will get the same chill tone. Well, who understands women anyway?
But I've expected code to be more friendly (it's mostly written by men after all).

OK, I get it, you supply me with a wonderfully written piece of code and if I'm unable to figure out the correct parameters or set the environment correctly then I'm the stupid one here and that's my problem to deal with. But really, isn't that what Error Handling is for?

Had a variety of error handling related issues lately, here are just some of them:

1. A few weeks ago I've tried to install an R12 Vision database. After copying the installation CD (the very first phase) the installer window have just vanished. The log did contain an error code but with no explanation whatsoever and I haven't found anything on Metalink, so I tried again and this time I saw that just before crashing the installer executes bootstrap.cmd. So I tried to run it myself and got a java error stack with an error message saying something about "Unable to open a window" (I don't really remember the exact formulation right now), with no other apparent choice I've started decompiling the java code together with my team leader (who used to be an Apps DBA as well). At last we've discovered a code section where an error wasn't handled, pleased with ourselves we've logged an SR.
An unhandled error is bad enough but apparently the real problem was that the Staging Area wasn't set correctly, that's definitely my bad but how complicated it is to check the validity of the Staging Area before launching the installer and throwing totally unrelated errors at me?

2. I've noticed that it doesn't matter what's wrong, if running adcfgclone on the database tier it will probably fail with the same error stating something about "Error returned while creating control files" and an RW-50004 error code. If you're lucky the log file will have some clues, if not you'll have to start debugging Oracle's code. I got this same error for problems with an Oracle script which is part of the whole process, for problems with starting the database listener and even for a corrupted database, non of which are really related specifically to the creation of control files. And I'm pretty sure this error shows itself in much more cases. I think it's totally reasonable to expect an error message when the problem appears, instead, all errors seem to propagate to a single line of code where the adcfgclone just won't take it anymore.

3. Had to modify the virtual IP(just the IP, host name stays the same) of my load balanced OID server - turns out that's not as simple as one would expect. From what I know up until now (haven't really resolved this issue yet) it seems that the IP is hardcoded in some configuration file or maybe even in the Meta-Repository database, that's odd. Maybe I'll write about it in another post when I'll have more idea. Anyway, in the process I've discovered a script that's supposed to do just that, it even has it's own folder directly under ORACLE_HOME. This fact made me suspicious at once since it makes this script look like a late add-on which wasn't considered with much thought. The code inside really stands up to the expectations, it seems no one have ever tested the code. It looks something like this:

...
if
echo bad thing 1 happened
goto end
...
if
echo bad thing 2 happened
goto end
...
...
:end
exit

Doesn't take a cmd expert to understand that running the script as it is won't show you any error messages (unless you're a really quick reader) and it will actually close your cmd window (why?!).

Well, I think that concludes my case.

Gotta Have Faith

2007-12-28T00:59:00.000+02:00

I'm an atheist.
But I really do understand the need to have something powerful and almighty to believe in. Especially since I'm a sysadmin. How come? Well, the way I see it, two of the reasons people invented god are:
1. People need something to depend on in hard times.
2. People need explanations for the unnatural.
Both apply to sysadmins.

I'll elaborate.

1. Counting on Luck
Sometimes life gets tough. Sometimes you have an issue with some installation that you can't solve (and from how things evolve you know the help won't come from support), you're way beyond schedule and more problems keep rushing at you.
In times like this I tend to tell myself, like a good friend of mine phrased a while ago, "In the very very end things will be OK". Actually, now that I read it, it sounds a bit pessimistic but the intention is actually the opposite, the "very very end" doesn't have to sound so fatal, it just means problems will be solved in their own time. Sometimes, one just have to realize that no matter how grim a situation is, the issue WILL be solved, there's really no way you're not going to succeed in upgrading the god-forsaken database! So why worry (too much)?
This approach might make me sound a bit of an indifferent person, but if that what keeps me going after spending a couple of weeks trying to solve an issue and not making any progress at all, I'll take the criticism.

2. Voodoo
People might raise an eyebrow reading about system administration and witchery in the same context but I think most of the people who deal with computers and software know that this stuff isn't really deterministic.
A good example is when a piece of code gives you a segmentation fault and after you put debug printouts in every possible line the segmentation fault just ceases to exist. Of course, after removing the printouts everything is fine, no segmentation faults.
An even better example will be what I refer to as the "Sysadmin Effect". Systems shouldn't be treated as anything less than an intelligent being with it's own will. It might be a bit difficult to take me seriously right now, but systems really need attention like little spoiled kids. They behave as long as the sysadmin is present but start revolting as soon as they're left alone.
The system might function perfectly for weeks and months but the second you leave the office for a two day trip a processor will get burned and the phenomenon will be so irrelevant that you won't have any chance of understanding the problem on the phone (and I'm talking from experience here).
Another one. A few weeks ago I went out in the middle of the day, when I returned and inspected the monitoring software I've discovered the system had a blocking sessions issue from the minute I went out... and up until a minute ago.
Those are just two examples, I have a lot more.
You might try to convince me that there's an explanation for everything and that there's no such thing as an intelligent system - and you'll probably be right.
But me, I just call it Voodoo.

All About Attitude

2007-12-17T22:23:00.000+02:00

In my previous post I've complained about being asked to perform major installations in order to solve small bugs.
So as I've checked the updates to my SR's coming to work after the weekend I had a classic example waiting for me. It's amazing how similar was the response to one of the SR's to mine made up example. Just proves (at least to myself, you haven't really seen the SR) I wasn't exaggerating:
"I've discovered a similar bug" (didn't look all that similar to me, though) "it has been solved in ATG_PF.H RUP5 but you might as well install RUP6".

BUT,

And you've probably noticed that's one hell of a BUT, the guy added "let me know if it's feasible".
Now, reading this have probably changed my reaction totally. Instead of getting furious at the support technician I just thought to myself: "The guy presented me with the option but he knows it's meaning and understands the implications of following this path".

Well, here is a lesson for everyone who provides support (and a sysadmin's job has definitely an aspect of support): Show the customer you care and that you understand his troubles.
It costs you nothing and you might actually get the other guy feeling a bit less frustrated.

Support, Yourself

2007-12-14T11:31:00.000+02:00

Good support is really important. I'm a member of a sysadmins' team and I know support can really suck.
The company that supports one of the products of a fellow sysadmin uses him for support. I mean seriously, how lame can support be if they send people down to us to understand how the system they support works?
Another company (and I'm talking about a major company here) seems to ignore a serious performance issue, the problem is present on their site too but they don't seem to be doing much towards supplying a solution.

Given those, I'm used to get lots of "You got a reply to your SR within 15 mins?! I wish I had at least someone to talk to about my issue...". And indeed, the Metalink and the support system of Oracle are most impressive at times.

But people are not really satisfied with what they have, are they? I'm no different.
I have the following rule of thumb:
When I open a new SR the outcome will be one of two:

1. It's a well known issue that I somehow managed to miss searching the Metalink, there's an easy solution, we all live happily ever after.

2. It's not a well known issue.
I will be asked to perform tests that are often irrelevant, this I can live with, after all who am I to really know the insides and decide what's relevant?
But the thing that really drives me mad is when I get something like the following
"Well, the issue is not reproducible on our test environment but I see you don't have any 11.5.9 CU installed and you're only on ATG_PF.H (no RUPs) so I'm gonna ask you to install to 11.5.9 CU2 and ATG_PF.H RUP6 to match our configuration. Oh, and in two days I will write that I haven't heard back from you and so I'll inactivate this SR in one day stating I believe the issue has been resolved by the trivial solution I provided. No, I don't really think your problem will be solved but you can't prove me wrong until you try and then you'll have your hands full dealing with the mess the recently released RUP6 has caused." Does the guy really know what he's talking about? Does he really think I'll perform a semi version upgrade to maybe solve a small issue?
So, in most cases, I might get some hints from support but I will eventually have to work out the solution on my own (or with help, but not from Oracle). I might even find a piece of bugged Oracle code (I'm on Windows so wrong direction slashes might cause problems, etc.) but then I'll get something like "You're right, our bad, here's an idea for a piece of code you can write to workaround the problem".

Another annoying thing is that in both cases I'll probably be asked some questions I've already answered. What's the point of asking me all those question when I open an SR if I'm going to be asked for my R12 Rapidwiz version again by the support technician? OK, I got another technician to handle my case, can't he read the SR and see what tests I've already performed?

Actually, this post isn't meant to be (maybe just a bit) about blowing out steam about how bad is Oracle's support, it's probably relatively good.
My point is that when it comes to the real stuff it's the sysadmin's job to solve the problem, he knows his system best and that's what he's paid for. Have all the support you want but some issues will only be solved after spending many days hacking the system and getting frustrated for not achieving anything and wasting your time on annoying problems. And all by yourself.

Finally, for all the support guys who come to read this post. Nothing personal, really. I'm sure you have complaints about us customers as well, so just take it easy :-).

Weber's Law

2007-12-07T10:26:00.000+02:00

I've started studying for my Masters in Computer Science this year and one of the courses I'm taking is Image Processing.
One of the early lectures was on color perception and so I was introduced to Weber's Law. In short, the law argues that perception of change depends on the relative change, for example, a candle lit in a dark room will probably be blinding, the same candle lit outside on a sunny day won't be noticed at all. Anyway, my explanation is probably a bit lame so just check out the link.
I always like to study about these things that show us how sophisticated human body (and nature in general) is, and how decades of research are spent to try and solve an issue that is already solved(we just have no idea how) in natural systems .

Few weeks later, I'm after a Judo session and my instructor tries to motivate one of the kids, he has to main points:
1. You're not supposed to suffer from practicing Judo, but on the other hand, you have to know what you want to achieve and realize you will have to put in some hard work to get there. Well, this is mostly a good approach to life but not what this post is about.
2. As you get better (the kid is no rookie) it's harder to tell you get better and this can sometimes be frustrating. Out of the scope and probably not subject to quantization but that's definitely a reincarnation of Weber's Law.

Actually, as I was listening to the conversation I thought it might as well be a discussion on relationships.
Only this week I realized the same applies for becoming a professional too. I recall that a month or so after I started doing my job on my own (not being a trainee anymore) I was excited to see how much I've learned and done in this relatively short period (and all - almost - on my own). Well, I'm definitely not there yet, I still learn new things daily and get thrilled by solving problems, but the sensation of progress is obviously not the same as at the beginning.
I think this is an interesting dilemma, anyone wants to be the top expert in his profession but once you get there, where's the excitement of learning new things and solving new issues?
Luckily, there are so many aspects in this line of work that one can always find something new and stimulating to learn, and this is what one should do if challenge and satisfaction is what he's after.

Personally, I think it's nearly impossible to become an expert of this caliber, but this Weber's Law stuff just got me thinking.

Default Configuration

2007-11-22T23:24:00.000+02:00

I always thought that default configuration should be the simplest and the easiest for implementation, well until I met Oracle default configurations.

A good demonstration of this will be describing two projects I worked on in the past year.

Some background before I begin. We've decided in our organization it's time to implement SSO for the EBS. Now, we already had the OID infrastructure required with an operable SSO configuration. It was, however, decided I should migrate the single OID server to two Load-Balanced OID servers, for High Availability purposes of course. So, first phase I migrate to Load Balanced OID servers, second phase I make the necessary EBS adjustments.
I was amazed in both phases with how much sorting out I needed to do to understand what I really need to implement.

The only documentation to work with to migrate to a Load Balanced OID layer is the documentation which shows how to migrate to a totally High-Availability configuration in each layer, so after printing out all the documentation, I had to get rid of all the RAC and such related parts and even after that there were many parts I didn't really need for the final migration procedure.

The documentation about migrating to SSO for the EBS is even more ambitious, for some reason it's assumed that the second I implement SSO I'll implement full synchronization between the EBS and the OID (probably involving the master source of identities as well). What's wrong with just using the Kerberos Ticket users get from Windows for EBS authentication? At least for beginners? In a TAR I've opened to solve some related issue I had real hard time explaining that I have no reason for registering OID since I don't want any synchronization. Actually, since it appears as such a trivial step in the installation process I had my own share of doubts on this point.

My real question is why every paper assumes I'm on RAC (although it WILL happen one of these days, probably)? What's wrong with describing basic configurations and supplying pointers to the more advanced options?

Here I am!

2007-11-16T14:01:00.000+02:00

Well, after a period of writing a blog inside my organization I've decided to broaden my audience.

A bit about myself. I've been an Oracle Application DBA for about a year and a half now, I also work with other (mostly Oracle) applications.

A bit about this blog. In my organizational blog I'm pretty limited in getting down to the real technical stuff I sometimes want to write about, from the simple reason that almost nobody will understand what I'm writing about.
It's actually pretty exciting, in my organizational blog I may pretend writing for some general population, but all in all I know 90 percent of my readers and work with them daily. With this blog on the contrary I don't have any idea who my readers will be and if I gonna have any at all. Well, gotta take the chance!

Well, so far the first post. As you've probably already understood the next posts gonna be more technical and given my expertise, probably will deal with Oracle (Applications) related stuff.