High-End Software

Like I mentioned a few months ago, I'm currently working on an Enterprise Search project. Since then I discovered a few other annoying things about FAST ESP and other high-end search solutions.

I've tested two main types of search solutions:

1. Entry level solutions - by companies that have another core business except search.

2. High-end solutions - by companies whose core business is search.

Obviously, there a lot of products somewhere in between, but that's always true. Actually, FAST has been lately acquired by Microsoft, but for the purpose of this post it is still under the second category.

Being a newcomer to this field of Enterprise Search I was quite innocent and thought that entry level solutions will be simple, basic and easy to use - no disappointment here, but I also expected high-end solutions to be a complete search suite that can do great and cool stuff.

Well, products I've examined can do great and cool stuff, I'll even exaggerate and say that it feels like everything is possible if you know your way around the product, but in no way these products I've examined are a suite. It almost seems (it's actually sounds like a swell business model) that the products are intentionally designed to make you grate your teeth at every step so they can provide their business partners with work for their consultants. It seems like they try to make the product as naked as possible, leaving only the basics of indexing efficiently, providing customizing tools and a few other abilities. As always, a whining is not complete without a few examples. I'll rely mostly on FAST ESP examples, but not only, I just know it the best:

1. I would expect a high-end solution to include some security enforcement. Say I want to index a file system(obviously there are other examples as well) content source, to the entry level products it's obvious that I want to index the ACLs as well, not so much to the high-end stuff. High-end software will require installing an additional module that I'll have to carefully configure. And that leads me straight to the next point...

2. To configure the security model for FAST ESP to enforce file system security I have to follow the documentation which, put in one simple word, sucks.

Fact: surprisingly the FAST guys realized I might want to index Windows based file systems.

What I'd expect to see in the documentation: a simple to-do list containing every step I should do in order to configure the whole thing.

What I actually got: every piece of the puzzle is in a different part of the documentation so before each of the following revelations I had to wonder why nothing works and why nothing is written (or at least not where I expect to find it):

• When the module is initially installed no indexed items are searchable, there are a few ways to work around it and they're written at the end of the documentation.
• There's no built in authentication module (a pity) but at least there are a few ways to work around it as well.
• To index securely you have to modify the processing pipeline (or the whatever the term the product uses), next point here I come.

3. The products I've examined have far too much of encoding related (Hebrew and computers should not co-exist) issues, some of them I've been able to overcome but still, for now I can't index securely paths with Hebrew in FAST ESP.

I have lots of other examples and I'll probably have a lot more as I go along, but I think I made my point. Totally high-end...

Development

I'm not a developer but as the team leader of the sysadmins (which is often referred to as the "technological team") in my department I am interested in the platforms and methods that are used for development, after all the bottom line is that I'm responsible for the stability of the systems and a system is more stable when the correct practices are used.

Let's take for example our Oracle ERP system (which I used to administer so I'm more familiar with it than with other systems). Today we have web applications based on different technologies:

1. PL/SQL cartridge based applications - well, the programmers do know PL/SQL so it's kinda OK, but no new applications are written using this technology and in the next EBS version (R12) it's no longer supported.

2. Java based applications - no one really knows them and they're supposed to be re-written.

3. ApEx based applications - small modifications are still made to them but the knowledge level is pretty low.

4. .NET based applications - that's the technology that all the developers are familiar with and with which the latest applications were developed.

We have all those techniques since each of them fitted a different need at a different time.

One technology we don't currently use is the Oracle Applications Framework which is a pretty strong technology but...

We currently have about 3.5 EBS developers none of which is really specialized in Java (and the Framework is Java based) and the turnover rate in the developers' team is pretty high.

These facts don't really get along well with the multiple development techniques we maintain.

So now it seems like the time to make a strategic decision as to how the EBS development will look in the next few years, I can think of two main approaches:

1. We choose one single technology that we'll use to write all the solutions, obviously it won't meet perfectly all the needs but the programmers will be specialized in this technology and training will be relatively simple.

2. We choose two technologies that will answer two major types of requirements - for example general user applications vs. specialized user applications. The programmers will be less specialized but we'll be able to provide better solutions.

Anything more than that will have too much training and maintenance overhead...

Disappointment

Sometimes you think you've just seen something amazing: on news, a discount or some great technology, since this blog has a certain orientation, I'll focus on the later.

This week I was planning my oncoming trip to Slovenia, and me and the Doc got to the point of looking at transportation, specifically trains.

Google led us to this site. If you enter the site you'll notice it adapts to your localization settings - nice! We've started typing "Venice" in the destination and not only we had auto completion we saw support for different writing options for "Venice" - really nice comparing to the dozens of other transportation sites we've seen (at some point we thought it could read Hebrew but that's really an exaggerated expectation). And now to the most amazing feature (tam tam tam...) the prices appear in local (ILS) coin! After being amazed for a couple of seconds (yes I know, it's not really that a sophisticated technology, but as mentioned above we did see other transportation sites), we've decided to check what the British version looks like.

Apparently the two sites give different prices for the exact same trip. Yeah, that's right, three adults traveling from Ljubljana to Venice buying tickets in Israel will pay 453 ILS and the same three adults will pay only 42 pounds traveling from Great Britain. European discount? Don't think so, since Australians pay less. I've tried long and hard to think what could be the code behind those ridiculous results, but I still have no idea, suggestions?

So, going back to the first paragraph the conclusion is that miracles do not exist, sorry kids.

Making Me Sweat

For quite some time now, I've been working on an Enterprise Search project for my organization. Last week I've installed a FAST ESP system as part of a POC, the system uses many third party software components and since each has it's own license they can't be provided as part of the ESP software and have to be downloaded as part of the installation process.

But, here's the catch, I don't have any Internet access on my network. The installation guide provides a solution for this kind of scenario - install the ESP on a computer with Internet access, after the components are downloaded you can abort the installation and transfer the components any way you want to your LAN.

This is a very strange solution, why run an unnecessary installation? What if I have limited privileges on my Internet computer?

After all, there's a much simpler solution - why not provide me with the links to the components? After all, what I did was to harvest the links from the manifest.xml the installation uses to download the components...

Why make me sweat?

Storming In Or Taking Baby Steps

When you have a problem (specifically, technology related) there are two main methods for dealing with it:

1. Storming In - throwing everything you have at the issue, e.g.: trying every performance related command you know, changing hardware, etc.

2. Taking Baby Steps - slowly and thoroughly analyzing the issue, e.g.: reproducing the issue on a development environment, checking all changes that have been made, etc.

Reality, like always, is some shade of gray so what you do in practice is usually some combination of the above two methods.

Many times it's not clear what approach to use. Storming in will sometimes solve the current crisis but might handle only the symptoms leaving you unprepared for the next time the problem manifests itself (and this time the instant solution might not work). Generally, it's always better to understand the causes for a problem and to investigate them, but sometimes issues won't reproduce on development environments and the causes are just too voodooish and some issues are just simpler to ignore. And after all, time IS a valuable resource.

The Stage I Fear The Most

Since I became a team leader I've been working more and more with Oracle products other than EBS, mainly iAS and OID. I've noticed that the installation part of iAS-like installations never fails, the stage I've really learned to hate is the Configuration Assistants - each one that completes gives me a sigh of relief, but too many times I'm cursing instead. 

For instance, lately I've been trying to upgrade my 10.2.0.2 OID instance to the latest 10.1.4.3 version, upgrade path looks like this: 10.2.0.2->10.1.4.0.1->upgrade MR->10.1.4.3.

10.1.4.0.1 - check. upgrade MR - check. 10.1.4.3 - uh oh...

Even before installation begins, Oracle tries to outsmart me - documentation says to shut down OPMN, but when I do that I get an error during the initial steps that says it can't determine running processes(like daaaa!). OK, so I start OPMN, but then I get a "dude, you're running services that bother me" message, "OK take it easy, I'm shutting it down" is my reply. Up until now I have the upper hand and the installation itself runs smoothly, as always.

But then comes the one before last CA - the DCM CA that fails because of a problem with an Apache dll. So after trying to solve it for quite some time (replaced dlls, configuration files, etc.) without any positive results I opened an SR, the action plan , is kinda funny - install 10.1.2.3 on top of 10.1.4.0.1, then try again, makes sense don't you think? Well, it does (a bit), my previous attempt did include a 10.1.2.3 upgrade before 10.1.4.01, some maybe I should've thought about it myself.

So I did just that, but then the SSO CA took ages (more like two hours, but you get the idea) to fail, some mixture of opmn stopall/startall got it through but now the OPMN startall CA fails so I didn't even got to the problematic DCM CA yet, I guess I have some frustrating days ahead...

With the CAs I always have the urge to just skip 'em all and run 'em later, on the other hand if things are wrong with the so basic OPMN startall maybe I should handle things now. I'll probably have more updates regarding this issue in following posts.

Linguistics Tip

OK, so we all know that when facing language issues, playing with NLS_LANG often does the trick, but a while ago we had an issue with a web app (running on iAS) that would show normally Hebrew characters for strings taken from configuration files but would transform any Hebrew data from the database into question marks. The interesting thing was that the same application running from a workstation worked perfectly. 

We were starting to get desperate, as more and more people looked at it and didn't manage to find a solution. But when I showed it to the DBA in my team he solved it in five minutes, luckily, he had a similar issue the week before.

Here's the trick, sometimes you have to modify the language settings for the server itself. Actually it's not enough to change the language settings, you have to check this annoying check box as well:

We've obviously played with the Regional Settings before just never even looked at this tab and hence didn't see any changes.