Linux System Freeze (1 Viewer)

[philipkw]

Over 10+ years I've had various kenel versions running on various hardware configurations with very little problem. I recently set up the following on 2 machines:

(1) Oracle Enterprise Linux (Redhat variant) Kernel 2.6.9-55.0.0.0.2.ELsmp on an AMD Athlon 64 Dual Core 4.2 Ghz processor with 1 gig RAM

(2) Oracle Enterprise Linux Kernel 2.6.9-67.0.0.0.1.EL on AMD Athlong 2.2 Ghz processor with 1 Gig RAM

These machines serve the following purpose:

Machine (1) runs Oracle Application Server 10gR2.
Machine (2) Runs Oracle 11g Database.

Since configuring these in January I've experienced systems freezes on both of them. The machine becomes totally non-responsive, isn't susceptible to a graceful shutdown using the Alt-SysRq-R-E-I-S-U-B key sequence and I'm left with no choice but the hard reset.

I can't find anything useful on Oracle's techsite or Google. As stated before I've had several installations of Linux running over the years, some much lesser hardware also running an Oracle database, with no significant issues.

No doubt the Oracle stuff is CPU/RAM intensive but I've got plenty of swap set up.

I suppose I could shutdown Oracle processes on both machines to see how long the machines will stay up. Before I did this I wanted to see if anyone had experienced similar issues with Linux. Plus since the freezes are intermittent and sometimes many days between occurences I'm not sure how much info I could glean from shutting down Oracle.

The only difference I can think of between my current installations and installations from prior years is the chipset. This is the first time I've installed Linux on AMD.

Anyone had experience with Linux on AMD? Or anyone exerpience similar behavior with Linux on any hardware platform?

TIA.

 

[robsmith32]

oracle 11g on 1g of ram? that's all i'm thinking, i know you mentioned a lot of swap.. but still.
can you use top or glance and see if using a ton of I/O during the time..

I know of incident with oracle 8, when and index got erased. the i/o went thru the roof and exhibited the same thing on much higher end machines.... took 6 hours for a gracefull shutdown.
thinking of that , make sure your querries are optimized so that not using a lot of io.

 

[philipkw]

I haven't monitored CPU or IO in depth. I've taken a look-see at top but nothing caught my eye. That's a good thought though. I probably should spend more time doing that.

I realize the machines are woefully underequipped to service a transactional DB. But they don't. There are no ongoing queries. These are simply machines I have at home to play with. I manage Oracle on Solaris and Linux all with 16g of RAM at work and the machines at home are simply testbeds. They mainly just set there chewing cycles with occasional DML.

Machine (2) which is the punier of the two has only experienced one freeze and that was last night. Interestingly enough it was after I installed at began running setiathome (in addition to the 11g database). Machine (1) has frozen several times.

I'm prone to think it's some sort of CPU and/or RAM overload issue but I'm not entirely convinced.

There are no core dumps and nothing written to /var/log/messages.

I ran 8i for many years on a *much* lesser machine with no issues. The difference is that machine ran Intel.

Thanks for the response.

 

[robsmith32]

lol, um, i shouldn't say this but i had eratic freezes for a little while on one of my amd builds.. wasn't paying attention and put the cpu fan on backwards... cpu overheat ..= immediate lockup
the difference being that the heat sink compound was only on half the actual chip instead of covering it.
live and learn.

setiathome, i'm not familiar with.
can't say i've ever ran oracle on amd.. only sun and intel, just thought that sounded too familiar of when a heavily used index got whiped to re-index it and decided cpu was too busy to do that. so then it had no index, therefore cpu and i/o went thru the roof.

 

[Buickman]

I was having a lot of lockups with Ubuntu Gutsy when I had it loaded on my Asus moboard that had the Nvidia i650 chipset. I swapped that moboard with a MSI D965 chipset and haven't had any problems, which leads me to believe that the Gutsy kernel didn't like something about nvidia. I had no problems running Feisty on that nvidia chipset. One thing I never did was try to build a new kernel with Gutsy. Maybe I should have. I haven't loaded any linux platform with an AMD cpu, only Intel.

I know that probably isn't much help, but maybe something to look at.

 

[Cajun Mike]

i am running gusty studio on an asus amd 64x2 nvidia 939 mobo w/nvidia graphics with no problems just answering buickman

 

[eaux-yeah]

Hey Philip.
Never messed around with Oracle Linux, basically a RHEL copy?

It sounds like a 'limits' or 'shared memory' problem. May need to manually set the hard and soft limits and/or tweak the SHM* settings?

I would consider 1Gb memory to be bare minimum for just about any operating system these days, real cheap, too. Increasing swap to compensate for lack of memory is a not the best approach. I would at least create the swap partition on a separate (and fast) drive. Once again, memory very cheap.

Using 'top' is just about the equivalent of your car's 'idiot lights', maybe a bit more informative, though.
I would recommend using 'vmstat', 'iostat', or better yet ........ DSTAT!

Do you see any strange information in the 'dmesg' output?
LSOF can help you find out if incorrect libraries or plug-ins, etc. are being used.

Hope that helps.

 

[eaux-yeah]

This might help, too. It's for RHEL, but should (hopefully) work for Oracle Linux, as well.
http://www.puschitz.com/TuningLinuxForOracle.shtml

 

[philipkw]

Hey Philip.
Never messed around with Oracle Linux, basically a RHEL copy?

It sounds like a 'limits' or 'shared memory' problem. May need to manually set the hard and soft limits and/or tweak the SHM* settings?

I would consider 1Gb memory to be bare minimum for just about any operating system these days, real cheap, too. Increasing swap to compensate for lack of memory is a not the best approach. I would at least create the swap partition on a separate (and fast) drive. Once again, memory very cheap.

Using 'top' is just about the equivalent of your car's 'idiot lights', maybe a bit more informative, though.
I would recommend using 'vmstat', 'iostat', or better yet ........ DSTAT!

Do you see any strange information in the 'dmesg' output?
LSOF can help you find out if incorrect libraries or plug-ins, etc. are being used.

Hope that helps.

Yeah, I think RH and Oracle have a partnership and Oracle tweaks the RH distro to customize for running Oracle.

I've got shared mem settings, limits, kernel parameters all set per the Oracle install guides. In fact, Oracle will not even install unless these parameters are set appropriately.

I'll check out those utilities you mention to see if I can discern anything.

Meanwhile, I've shut the Oracle app server down on machine (1) as an experiment to see if the system stays up. At this point, I can't be convinced it's even Oracle-related without more data points.

The other thing I failed to mention is that both machines are on wireless LAN. I had some headaches getting that going using ndiswrapper and wpasupplicant. I'm new to the wireless thing (just set up my wireless LAN at the same time that I built these 2 machines) and have been disappointed to see that these machines sometimes lose connectivity. I scripted cron jobs for both systems to poll the network to see if it's still up. If not, the script restarts the network. I manage my entire LAN via wireless laptop and have also noticed that the servers are occassionaly sluggish/slow to respond. I have no idea of the wireless woes are contributing to the freeze. Probably not.

I also hacked my router and installed the DD-WRT firmware which allows increasing the radio transmission in an effort to address this situation. I might need to revisit that. ??

Much like the 1 gig of RAM, I realize that wireless isn't ideal for running a production system, but it's not. It's serving just me. I'm just a hacker that likes to see what I can build/tweak around the house.

Thanks for all the info!