Good Problems

Most of my posts have something to do with complaining about different products and the ridiculous bugs they have, and anyway who loves problems? Every sysadmin would like to come to the office in the morning drink his cup of coffee (or tea, in my case) and go spend the rest of the day at the beach knowing everything will work perfectly and his presence won't be needed.

But problems, as strange as it sounds, have an upside as well.

First of all, if things were so simple, people like me would be unemployed. We'd still be able to go to the beach, but we wouldn't be getting paid for it. But even without fantasizing about a prefect world, problems are important. Might sound like I'm a bit exaggerating, but I really think problems make you a better sysadmin.

Come to think about it, problems to a sysadmin are like fights to a relationship, no one really wakes up in the morning and thinks to himself "Oh, I'd like a good argument today", but couples do fight and resolving those arguments usually strengthens the bond (and like with systems too much fighting is probably a bad sign). And like those arguments, each problem you tackle makes you understand your system a bit better, understanding that is hard to achieve when everything just works.

The ERP system I used to administer is a great example. ERPs tend to be complex systems and ours not different, we have many utility scripts that accompany the system. Sure, I knew there were some scripts responsible for printing, but until I encountered some problems with them I didn't really get to understand each and single line of code in those scripts. And on the other hand, I wrote some scripts for an interface a few years ago, and those scripts work almost perfectly (I'm not necessarily objective here) up until now, so I'm pretty sure my replacement doesn't really know what's going on inside.

So maybe I should start giving more respect and appreciation to the problems I encounter (and complain less).

Pessimist

Generally, I think in most cases we all should be a bit pessimistic. Being objective is hard and personally I'd rather have a nice surprise than a great disappointment. But that's arguable - I guess optimistic people live longer - but then again, the benefit of long life is arguable as well.

But what I think is not really arguable, is that in my line of work you have to be pessimistic. A major part of being a sysadmin is keeping the system stable, and you can't really do this by being optimistic. To make the system stable you try to think about any scenario as improbable(and pessimistic) as it might be. One good example I can think of, is that having a daily backup is very important in our organization(and I really do hope we're not the only ones), yet for most systems, we've never needed those backups.

I might be taking this a bit too far when I'm berating team members who behave "too optimistically" and tend too answer "I'm getting paid for it" when asked why I'm so pessimistic, but I'm really just trying to make my point.

Fixation

Being a sysadmin has many controversies, take new technologies for instance:

Recently I've installed a Weblogic server to start examining it, as our Oracle AS servers will need to be replaced sometime in the next couple of years since this line of products is going bye-bye. Anyways, it was pretty exciting, new (for me) Fusion 11g UI and the main administration screen is SO busy. So it sounds like I'm all for new technologies... not really, not always.

Yes, you should upgrade existing technologies. Yes, you should check what new technologies are out there. No, you shouldn't use a new technology where an existing one could be used.

I think the best example for my point is building solutions for specific needs.

Let's take my organization's Oracle ERP system for example. Many times a specific need arises and people suggest all kind of cool solutions. In my opinion it's very important to try and base such a solution on an existing technology, I'd rather have a solution that is not perfect than a perfect(like such things exist) solution that will handle this specific case alone and that no one will know about if something goes wrong a few years from now.

So, sometimes, I end up being this guy with a fixation on old and boring technologies...

And if talking about sysadmins, here is a good one :).

Roach-Tech

When a company designs a product, there are many aspects to take into consideration. The product should be appealing to the eye, ergonomic, cheap to produce, etc.. One aspect I believe most vendors don't consider is that the product should be roach unfriendly. What am I talking about?

Well, take for example Avaya's phone:

You can see that attached to it's back (didn't find a better image) is a panel that gives it it's angle. Apparently this panel is a great habitat (at least in the Israeli climate) for roaches and they even consider it a great place to drop eggs:

That's right, this brown thing is a roach egg.

The phone doesn't look so attractive right now, does it?

Randomness

Time for a break from professional stuff. The only thing common to this post and my professional posts is my tendency for voodoo crap (although I do consider myself a very reasonable guy).

Lately I've been subject to several events that made me wonder if everything is really a coincidence. What's more interesting is that those different kind of events are not something I've invented and I know many people (supporting comments please) who have experienced similar things as well (and that's not the first time I experience this kind of events as well):

1. Useful Fact - Everyone has some relatively known facts that he's missed during his life, me included. What is strange is that many times you learn one of those facts and then it suddenly pops up everywhere. My example from this month: a lecturer was giving a probability example and mentioned the - unknown to me before - fact that pingvins do not live near the northern pole, a day later this fact was mentioned in a movie.

2. Lucky Instinct - A few weeks ago I was checking out Google Dashboard and saw that my last chat conversation has been with someone I knew I didn't talk in a while. I started constructing my next blog post about how Google showed me totally wrong data, but decided to check my "All Mail" anyway. To my surprise I did have a conversation with this person. Apparently she sent me a message just as I closed GMail and unlike in other times when I miss a chat, GMail didn't put this conversation in my Inbox so I had no real way of knowing this message was sent(that's actually a GMail bug as well, but not as bad as showing totally wrong data). This person wished me a happy holiday so this coincidence saved me from being rude. This whole story is a bit like when you forget something minor at home, come back and then recall you actually forgot something much more important and start wondering what would've happened if you didn't recall about the minor thing. But here you can give credit to subconscious, unlike in my example where I just happen to check the Dashboard on the same day. On the other hand, I might have missed a lot of other chats.

3. Déjà vu month - During life you meet and interact with many people and with most of them you later lose contact, you might occasionally meet one of them but that's it. So it strikes me very unlikely that during the last month I've met\reconnected with three different people (with whom I had significant interaction before) from my past. The average amount of time I didn't have any contact with those people is 4 years! And no, those people don't have anything at all in common and don't know one another. I actually had similar month a couple of years ago (interestingly - and unlikely - I think it was the same time of the year).

Do you believe in voodoo now?

Business Model

Every company has a business model. It can focus on selling a product, selling accessories for the product, selling support, etc.

One type of model is the "consultation" model, it's used when the product the company sells is sold for near to nothing, but the company still has to make money somehow, so the company presents you with experts that will help you through every phase of the implementation.

Now there are two options - the consultant gets paid either on a per project basis or on a per hour basis. The per project option is OK, but it's highly unprofitable for the company - it's never a good idea to commit yourself to an amount of work you don't know how to translate into time(==money).

The per hour option is problematic from the client's side. A "good" consultant from the company's perspective is one that can do the same work for the longest amount of time - to me it seems to contradict professionalism, but that's not my point here - that kind of consultant has no motivation to do his work to the point and as effective as possible or even answer questions on the phone - every question can be seen as something that should be thoroughly discussed on the client's site (and expense).

Now, with support things are easier - if a client feels dealing with a support issue is taking too long he concludes there's a problem with the company, and that's obviously not so good for the company. But consulting taking a long time can easily be painted like a good and thorough work on behalf of the consultant, and since the client often really doesn't have a good idea about the product he can be easily fooled.

Another pitfall you should beware of when taking on a software project.

High-End Software

Like I mentioned a few months ago, I'm currently working on an Enterprise Search project. Since then I discovered a few other annoying things about FAST ESP and other high-end search solutions.

I've tested two main types of search solutions:

1. Entry level solutions - by companies that have another core business except search.

2. High-end solutions - by companies whose core business is search.

Obviously, there a lot of products somewhere in between, but that's always true. Actually, FAST has been lately acquired by Microsoft, but for the purpose of this post it is still under the second category.

Being a newcomer to this field of Enterprise Search I was quite innocent and thought that entry level solutions will be simple, basic and easy to use - no disappointment here, but I also expected high-end solutions to be a complete search suite that can do great and cool stuff.

Well, products I've examined can do great and cool stuff, I'll even exaggerate and say that it feels like everything is possible if you know your way around the product, but in no way these products I've examined are a suite. It almost seems (it's actually sounds like a swell business model) that the products are intentionally designed to make you grate your teeth at every step so they can provide their business partners with work for their consultants. It seems like they try to make the product as naked as possible, leaving only the basics of indexing efficiently, providing customizing tools and a few other abilities. As always, a whining is not complete without a few examples. I'll rely mostly on FAST ESP examples, but not only, I just know it the best:

1. I would expect a high-end solution to include some security enforcement. Say I want to index a file system(obviously there are other examples as well) content source, to the entry level products it's obvious that I want to index the ACLs as well, not so much to the high-end stuff. High-end software will require installing an additional module that I'll have to carefully configure. And that leads me straight to the next point...

2. To configure the security model for FAST ESP to enforce file system security I have to follow the documentation which, put in one simple word, sucks.

Fact: surprisingly the FAST guys realized I might want to index Windows based file systems.

What I'd expect to see in the documentation: a simple to-do list containing every step I should do in order to configure the whole thing.

What I actually got: every piece of the puzzle is in a different part of the documentation so before each of the following revelations I had to wonder why nothing works and why nothing is written (or at least not where I expect to find it):

• When the module is initially installed no indexed items are searchable, there are a few ways to work around it and they're written at the end of the documentation.
• There's no built in authentication module (a pity) but at least there are a few ways to work around it as well.
• To index securely you have to modify the processing pipeline (or the whatever the term the product uses), next point here I come.

3. The products I've examined have far too much of encoding related (Hebrew and computers should not co-exist) issues, some of them I've been able to overcome but still, for now I can't index securely paths with Hebrew in FAST ESP.

I have lots of other examples and I'll probably have a lot more as I go along, but I think I made my point. Totally high-end...