Disappointment

Sometimes you think you've just seen something amazing: on news, a discount or some great technology, since this blog has a certain orientation, I'll focus on the later.

This week I was planning my oncoming trip to Slovenia, and me and the Doc got to the point of looking at transportation, specifically trains.

Google led us to this site. If you enter the site you'll notice it adapts to your localization settings - nice! We've started typing "Venice" in the destination and not only we had auto completion we saw support for different writing options for "Venice" - really nice comparing to the dozens of other transportation sites we've seen (at some point we thought it could read Hebrew but that's really an exaggerated expectation). And now to the most amazing feature (tam tam tam...) the prices appear in local (ILS) coin! After being amazed for a couple of seconds (yes I know, it's not really that a sophisticated technology, but as mentioned above we did see other transportation sites), we've decided to check what the British version looks like.

Apparently the two sites give different prices for the exact same trip. Yeah, that's right, three adults traveling from Ljubljana to Venice buying tickets in Israel will pay 453 ILS and the same three adults will pay only 42 pounds traveling from Great Britain. European discount? Don't think so, since Australians pay less. I've tried long and hard to think what could be the code behind those ridiculous results, but I still have no idea, suggestions?

So, going back to the first paragraph the conclusion is that miracles do not exist, sorry kids.

Making Me Sweat

For quite some time now, I've been working on an Enterprise Search project for my organization. Last week I've installed a FAST ESP system as part of a POC, the system uses many third party software components and since each has it's own license they can't be provided as part of the ESP software and have to be downloaded as part of the installation process.

But, here's the catch, I don't have any Internet access on my network. The installation guide provides a solution for this kind of scenario - install the ESP on a computer with Internet access, after the components are downloaded you can abort the installation and transfer the components any way you want to your LAN.

This is a very strange solution, why run an unnecessary installation? What if I have limited privileges on my Internet computer?

After all, there's a much simpler solution - why not provide me with the links to the components? After all, what I did was to harvest the links from the manifest.xml the installation uses to download the components...

Why make me sweat?

Storming In Or Taking Baby Steps

When you have a problem (specifically, technology related) there are two main methods for dealing with it:

1. Storming In - throwing everything you have at the issue, e.g.: trying every performance related command you know, changing hardware, etc.

2. Taking Baby Steps - slowly and thoroughly analyzing the issue, e.g.: reproducing the issue on a development environment, checking all changes that have been made, etc.

Reality, like always, is some shade of gray so what you do in practice is usually some combination of the above two methods.

Many times it's not clear what approach to use. Storming in will sometimes solve the current crisis but might handle only the symptoms leaving you unprepared for the next time the problem manifests itself (and this time the instant solution might not work). Generally, it's always better to understand the causes for a problem and to investigate them, but sometimes issues won't reproduce on development environments and the causes are just too voodooish and some issues are just simpler to ignore. And after all, time IS a valuable resource.

The Stage I Fear The Most

Since I became a team leader I've been working more and more with Oracle products other than EBS, mainly iAS and OID. I've noticed that the installation part of iAS-like installations never fails, the stage I've really learned to hate is the Configuration Assistants - each one that completes gives me a sigh of relief, but too many times I'm cursing instead. 

For instance, lately I've been trying to upgrade my 10.2.0.2 OID instance to the latest 10.1.4.3 version, upgrade path looks like this: 10.2.0.2->10.1.4.0.1->upgrade MR->10.1.4.3.

10.1.4.0.1 - check. upgrade MR - check. 10.1.4.3 - uh oh...

Even before installation begins, Oracle tries to outsmart me - documentation says to shut down OPMN, but when I do that I get an error during the initial steps that says it can't determine running processes(like daaaa!). OK, so I start OPMN, but then I get a "dude, you're running services that bother me" message, "OK take it easy, I'm shutting it down" is my reply. Up until now I have the upper hand and the installation itself runs smoothly, as always.

But then comes the one before last CA - the DCM CA that fails because of a problem with an Apache dll. So after trying to solve it for quite some time (replaced dlls, configuration files, etc.) without any positive results I opened an SR, the action plan , is kinda funny - install 10.1.2.3 on top of 10.1.4.0.1, then try again, makes sense don't you think? Well, it does (a bit), my previous attempt did include a 10.1.2.3 upgrade before 10.1.4.01, some maybe I should've thought about it myself.

So I did just that, but then the SSO CA took ages (more like two hours, but you get the idea) to fail, some mixture of opmn stopall/startall got it through but now the OPMN startall CA fails so I didn't even got to the problematic DCM CA yet, I guess I have some frustrating days ahead...

With the CAs I always have the urge to just skip 'em all and run 'em later, on the other hand if things are wrong with the so basic OPMN startall maybe I should handle things now. I'll probably have more updates regarding this issue in following posts.

Linguistics Tip

OK, so we all know that when facing language issues, playing with NLS_LANG often does the trick, but a while ago we had an issue with a web app (running on iAS) that would show normally Hebrew characters for strings taken from configuration files but would transform any Hebrew data from the database into question marks. The interesting thing was that the same application running from a workstation worked perfectly. 

We were starting to get desperate, as more and more people looked at it and didn't manage to find a solution. But when I showed it to the DBA in my team he solved it in five minutes, luckily, he had a similar issue the week before.

Here's the trick, sometimes you have to modify the language settings for the server itself. Actually it's not enough to change the language settings, you have to check this annoying check box as well:

We've obviously played with the Regional Settings before just never even looked at this tab and hence didn't see any changes.

Linguistics

My working assumption always was that about 33% of my duty as an administrator is language related, it might sound like a hell of an exaggeration but experience shows I'm not that far off target. You can always count on Hebrew to supply some issue with printing, badly encoded databases or question marks instead of readable text. The problem with linguistics' issues is that they tend to be hard to solve, you would expect most of these problems to have a common root cause but the fact is that a new solution is required each time, you do collect useful tips as time goes by but there's always some new issue waiting around the corner.

Hack, if I think about what I did for the last month it seems like most of the time spent was somehow connected to the fact that English is not the only existing language, I even think I have material for a couple of posts with useful tips and a lot of desperation.

Usually when I encounter these issues I blame Hebrew for being so odd - wrong writing direction and weird characters don't make my job easier, but English seems to be pretty fucked up as well. Sometimes I wish I could find the guy who invented case (as in lower case and upper case), it really makes no sense, OK so we'd write "english" instead of "English", who cares!

About a year ago I've mentioned researching EUS, well, we managed to implement (the rather cool) design we wanted but not without trouble. 

As part of the design we wanted to run command line esm commands to assign global roles to enterprise roles, the problem was that whenever we added a global role with the command line tool the whole list of global roles would just disappear from the GUI. How come you ask, and case sensitivity I answer! The issue was that the record inserted into the OID had case issues. It took me a few months to stop waiting for Oracle's solution and a few days to decompile the code and find the problem, it took Oracle another couple of months to supply a working patch.

And all of the this because of that stupid case invention.

Security Issues

A while ago I was phoned by the Apps DBA when I was away from the office and he told me the production is experiencing an issue that includes concurrents failing with an error message regarding access problems to .tmp files. It's not the first time I encounter this kind of problems and it's needles to say that it was never the stated issue with file permissions or disk space. So I came to the office to help the DBA to solve this issue (nothing trivial worked).

Solving the problem was odd, I had a good guess regarding the solution based on the previous cases, but it's really strange it did work since if what I did fixed a real problem I have no idea what triggered it and how the issue didn't manifest itself before, since nothing I know of changed (definitely not the problematic database parameter). Concurrents just started crashing with no apparent reason.

But that's not the main issue here (unless you're stuck with it and then you better know the solution or else you'll have some "fun" trying to solve the issue).

The really alarming thing about this case is that when concurrents crashed some of them had the apps password  (in cleartext) in their log files - the same log files every user can see for the concurrents they submit. 

Really, I couldn't believe my eyes, it seemed like the format usually present in those damn .tmp files, but I really don't understand what kind of a leak can cause the apps password to be dumped into the log file, to tell the truth it almost looks like an intended (buffer overflow?) attack.

The implications are very unnerving, what if we didn't happen to check a problematic log (not all of them had the issue) or what if we had just overlooked it? The meaning would be that everyone would be able to see the password until those logs were purged and we wouldn't have a clue. How can we be sure it won't happen again and who guarantees we will notice it the next time?