Segmentation fault (Apache 1.3 & 2) when trying to login to OTRS

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I have google'd, and searched the mailing list archives for similar
problems - but I can't find anything, so here goes...
I've been trying to get OTRS to work for days, with FreeBSD 4.9-STABLE and
both apache 2.0.49 / modperl 1.99_14 and apache 1.3.31 / modperl 1.29.
Whatever I do I can get to the login screen of OTRS but as soon as I login
apache segfaults:
[Fri Jun 11 14:02:58 2004] [info] Server: Apache/2.0.49, Interface: mod_ssl/2.0.49, Library: OpenSSL/0.9.7d
[Fri Jun 11 14:02:58 2004] [info] mod_unique_id: using ip addr 62.3.218.18
[Fri Jun 11 14:02:59 2004] [notice] Digest: generating secret for digest authentication ...
[Fri Jun 11 14:02:59 2004] [notice] Digest: done
[Fri Jun 11 14:02:59 2004] [info] Init: Initializing OpenSSL library
[Fri Jun 11 14:02:59 2004] [info] Init: Seeding PRNG with 0 bytes of entropy
[Fri Jun 11 14:02:59 2004] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Fri Jun 11 14:02:59 2004] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Fri Jun 11 14:02:59 2004] [info] Init: Initializing (virtual) servers for SSL
[Fri Jun 11 14:02:59 2004] [info] Server: Apache/2.0.49, Interface: mod_ssl/2.0.49, Library: OpenSSL/0.9.7d
[Fri Jun 11 14:02:59 2004] [notice] Apache/2.0.49 (FreeBSD) mod_perl/1.99_14 Perl/v5.8.4 mod_ssl/2.0.49 OpenSSL/0.9.7d configured -- resuming normal operations
[Fri Jun 11 14:02:59 2004] [info] Server built: Jun 11 2004 13:46:20
[Fri Jun 11 14:02:59 2004] [debug] prefork.c(955): AcceptMutex: flock (default: flock)
[Fri Jun 11 14:03:54 2004] [notice] child pid 79346 exit signal Segmentation fault (11)
[Fri Jun 11 14:04:00 2004] [notice] child pid 79347 exit signal Segmentation fault (11)
Jun 11 14:03:54 solaris OTRS-CGI-90[79346]: [Notice][Kernel::System::Auth::DB::Auth] User: root@localhost logged in
(REMOTE_ADDR: 62.3.218.17).
Jun 11 14:03:54 solaris pid 79346 (httpd), uid 80: exited on signal 11
Jun 11 14:04:00 solaris OTRS-CGI-90[79347]: [Notice][Kernel::System::Auth::DB::Auth] User: root@localhost logged in
(REMOTE_ADDR: 62.3.218.17).
Jun 11 14:04:00 solaris pid 79347 (httpd), uid 80: exited on signal 11
I've set apache's logging to debug but have found no further clue to what
the problem is.
In addition, while the apache-perl-startup script worked with apache
1.3.31, the startup script for apache2 does not work:
solaris# apachectl start
[Fri Jun 11 13:57:28 2004] [error] GATEWAY_INTERFACE not Perl! at /opt/otrs/scripts/apache2-perl-startup.pl line 5.\nCompilation failed in
require at (eval 4) line 1.\n
[Fri Jun 11 13:57:28 2004] [error] Can't load Perl file: /opt/otrs/scripts/apache2-perl-startup.pl for server
solaris.silverdream.org:80, exiting...
Although I realise this is not required for OTRS to run, I'm grasping at
straws trying to figure out what's going wrong beyond a blanket segfault.
Apache segfaulting each time points to a problem with OTRS but I still
can't find any log entries (I checked in /opt/otrs/var/log/...) to suggest
what is going wrong...
Any help would be really appreciated, specifically, are there any debug
options for otrs, I checked the docs but that section has yet to be
written :)
Thanks in advance,
- --
- -jamie

On Friday, June 11, 2004 3:14 PM
Jamie L. Penman-Smithson
[Fri Jun 11 13:57:28 2004] [error] GATEWAY_INTERFACE not Perl! at /opt/otrs/scripts/apache2-perl-startup.pl line 5.\nCompilation failed in require at (eval 4) line 1.\n
Comment line 5 in apache2-perl-startup.pl, and add these lines after line 5: # check $ENV{MOD_PERL}, $ENV{GATEWAY_INTERFACE} is deprecated for mod_perl2 $ENV{MOD_PERL} =~ /mod_perl/ or die "MOD_PERL not used!"; Now try again.
[Fri Jun 11 13:57:28 2004] [error] Can't load Perl file: /opt/otrs/scripts/apache2-perl-startup.pl for server solaris.silverdream.org:80, exiting...
Although I realise this is not required for OTRS to run, I'm grasping at straws trying to figure out what's going wrong beyond a blanket segfault. Apache segfaulting each time points to a problem with OTRS but I still
Where is your Perl located? Isn't that at a different location than /usr/bin/perl, say: /usr/local/bin/perl? If so, you should create a link in /usr/bin, pointing to the real location. Alternatively, you may change every bingbang line. These are about 60.
can't find any log entries (I checked in /opt/otrs/var/log/...) to suggest what is going wrong...
Logging goes to syslog usually. These options (=> Config.pm) let you basically change this: # (log backend module) # $Self->{LogModule} = 'Kernel::System::Log::SysLog'; $Self->{LogModule} = 'Kernel::System::Log::File'; # param for LogModule Kernel::System::Log::SysLog # $Self->{'LogModule::SysLog::Facility'} = 'user'; # param for LogModule Kernel::System::Log::File (required!) $Self->{'LogModule::LogFile'} = '/tmp/otrs.log';
Any help would be really appreciated, specifically, are there any debug options for otrs, I checked the docs but that section has yet to be written :)
You may switch on Debugging in many modules. If it would make sense (ie. if it were implemented in the module's code), you will find s.th. like the following towards the head of the file bearing the module: $Self->{Debug} = 0; Atm, this is true for these files: $ grep -lr "\$Self->{Debug} = [0-9]" . ./Kernel/Config/Defaults.pm ./Kernel/Config/.Defaults.pm ./Kernel/Language.pm ./Kernel/Output/HTML/Generic.pm ./Kernel/System/CheckItem.pm ./Kernel/System/Auth/DB.pm ./Kernel/System/Auth/LDAP.pm ./Kernel/System/Auth/HTTPBasicAuth.pm ./Kernel/System/Encode.pm ./Kernel/System/Spelling.pm ./Kernel/System/AuthSession/DB.pm ./Kernel/System/AuthSession/FS.pm ./Kernel/System/AuthSession/IPC.pm ./Kernel/System/CustomerAuth/DB.pm ./Kernel/System/CustomerAuth/LDAP.pm ./Kernel/System/CustomerAuth/HTTPBasicAuth.pm ./Kernel/System/Time.pm ./Kernel/Config.pm Or even: $Self->{Debug} = $Param{Debug} || 0; This instructs the module to inherit the setting from Config.pm. These do so: $ grep -lr "\$Self->{Debug} = [^0-9]" . ./Kernel/Modules/Not4Me.pm ./Kernel/System/DB.pm ./Kernel/System/EmailParser.pm ./Kernel/System/PostMaster.pm ./Kernel/System/Ticket.pm ./Kernel/System/PostMaster/Filter/Match.pm ./Kernel/System/PostMaster/Filter/CMD.pm ./Kernel/System/PostMaster/Filter/MatchDBSource.pm ./Kernel/System/PostMaster/DestQueue.pm ./Kernel/System/PostMaster/FollowUp.pm ./Kernel/System/PostMaster/NewTicket.pm As you see above, you may set this paramter in Config.pm to be valid for a numerous modules. But watch out, enabling Debug in Config.pm may lead to the server no longer responding due to the enormous amount of data being logged then. So it's safest to enable Debugging per module (and switch it off when done, btw). You may use any value above 0 - the higher the value, the higher the amount of information. Most modules do only ask for having Debug > 0, but it won't be unlikely we implement some kind of levelling someday. So using a value of 9 should always show the maximum atow. I hope this helps a bit, Robert Kehl -- ((otrs.de)) :: OTRS GmbH :: Norsk-Data-Str. 1 :: 61352 Bad Homburg http://www.otrs.de/ :: Tel. +49 (0)6172 4832388

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [Sorry for the long length of this, I've tried to cut down on the log snippets where possible...] On Fri, 11 Jun 2004 17:16:45 +0200, Robert Kehl scribbled down: <snip error message>
Comment line 5 in apache2-perl-startup.pl, and add these lines after line 5: <snip> Now try again.
The perl startup script works fine now. That's one problem solved, at least.
Where is your Perl located? Isn't that at a different location than /usr/bin/perl, say: /usr/local/bin/perl? If so, you should create a link in /usr/bin, pointing to the real location. Alternatively, you may change every bingbang line. These are about 60.
It's located at /usr/local/bin/perl, however there is a symlink from /usr/bin/perl to /usr/local/bin/perl so this shouldn't cause any problems.
Any help would be really appreciated, specifically, are there any debug options for otrs, I checked the docs but that section has yet to be written :)
You may switch on Debugging in many modules. If it would make sense (ie. if it were implemented in the module's code), you will find s.th. like the following towards the head of the file bearing the module:
$Self->{Debug} = 0;
I've turned on debugging for all modules and still I can't see anything which would explain what's happening... [Fri Jun 11 18:59:09 2004][Debug][ModPerl::ROOT::ModPerl::Registry::opt_otrs_bin_cgi_2dbin_index_2epl::handler][80] Global handle started... [Fri Jun 11 18:59:09 2004][Debug][Kernel::Language::new][82] UserLanguage = en [Fri Jun 11 18:59:09 2004][Debug][Kernel::Language::new][92] Kernel::Language::en load ... done. [Fri Jun 11 18:59:09 2004][Debug][Kernel::Language::Get][188] ->Get('Fri') Is not translated!!! [Fri Jun 11 18:59:09 2004][Debug][Kernel::Language::Get][188] ->Get('Jun') Is not translated!!! [Fri Jun 11 18:59:09 2004][Notice][Kernel::System::Auth::DB::Auth] User: 'root@localhost' tried to login with Pw: 'root' (1/roK20XGbWEsSM/roK20XGbWEsSM/ro/62.3.218.17) [Fri Jun 11 18:59:09 2004][Notice][Kernel::System::Auth::DB::Auth] User: root@localhost logged in (REMOTE_ADDR: 62.3.218.17). In /var/log/messages: Jun 11 18:54:58 solaris pid 82541 (httpd), uid 80: exited on signal 11 Jun 11 18:54:58 solaris pid 83315 (httpd), uid 80: exited on signal 11 Jun 11 18:59:09 solaris pid 83269 (httpd), uid 80: exited on signal 11 ...and httpd-error.log seems to suggest that I'm missing some config parameters - this doesn't look as though it'd cause OTRS to segfault though? Error: Config.pm No value for 'LogModule::LogFile::Date' in Config.pm found! Error: Config.pm No value for 'LogSystemCacheSize' in Config.pm found! Error: Config.pm No value for 'Database::Limit' in Config.pm found! Error: Config.pm No value for 'Database::DirectBlob' in Config.pm found! Error: Config.pm No value for 'Database::Attribute' in Config.pm found! Error: Config.pm No value for 'Database::QuoteSingle' in Config.pm found! Error: Config.pm No value for 'Database::QuoteBack' in Config.pm found! Error: Config.pm No value for 'User::PreferencesModule' in Config.pm found! Error: Config.pm No value for 'User::PreferencesModule' in Config.pm found! Error: Config.pm No value for 'CustomerUser1' in Config.pm found! Error: Config.pm No value for 'CustomerUser2' in Config.pm found! Error: Config.pm No value for 'CustomerUser3' in Config.pm found! Error: Config.pm No value for 'CustomerUser4' in Config.pm found! Error: Config.pm No value for 'CustomerUser5' in Config.pm found! Error: Config.pm No value for 'CustomerUser6' in Config.pm found! Error: Config.pm No value for 'CustomerUser7' in Config.pm found! Error: Config.pm No value for 'CustomerUser8' in Config.pm found! Error: Config.pm No value for 'CustomerUser9' in Config.pm found! Error: Config.pm No value for 'CustomerUser10' in Config.pm found! Error: Config.pm No value for 'SendmailBcc' in Config.pm found! Error: Config.pm No value for 'CustomerUser1' in Config.pm found! Error: Config.pm No value for 'CustomerUser2' in Config.pm found! Error: Config.pm No value for 'CustomerUser3' in Config.pm found! Error: Config.pm No value for 'CustomerUser4' in Config.pm found! Error: Config.pm No value for 'CustomerUser5' in Config.pm found! Error: Config.pm No value for 'CustomerUser6' in Config.pm found! Error: Config.pm No value for 'CustomerUser7' in Config.pm found! Error: Config.pm No value for 'CustomerUser8' in Config.pm found! Error: Config.pm No value for 'CustomerUser9' in Config.pm found! Error: Config.pm No value for 'CustomerUser10' in Config.pm found! Error: Config.pm No value for 'Sendmail' in Config.pm found! Error: Config.pm No value for 'SendmailBcc' in Config.pm found! The last several lines are: Debug: Config.pm ->Get('TicketNumberGenerator') --> Kernel::System::Ticket::Number::DateChecksum Debug: Config.pm ->Get('TicketIndexModule') --> Kernel::System::Ticket::IndexAccelerator::RuntimeDB Debug: Config.pm ->Get('TicketStorageModule') --> Kernel::System::Ticket::ArticleStorageDB Debug: Config.pm ->Get('Sendmail') --> Error: Config.pm No value for 'Sendmail' in Config.pm found! Debug: Config.pm ->Get('SendmailBcc') --> Error: Config.pm No value for 'SendmailBcc' in Config.pm found! Debug: Config.pm ->Get('FQDN') --> support.pinklemon.net Debug: Config.pm ->Get('Organization') --> PinkLemon Internet Services Debug: Config.pm ->Get('ArticleDir') --> /opt/otrs/var/article Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Debug: Config.pm ->Get('AuthModule') --> Kernel::System::Auth::DB Debug: Config.pm ->Get('DatabaseUserTable') --> system_user Debug: Config.pm ->Get('DatabaseUserTableUserID') --> id Debug: Config.pm ->Get('DatabaseUserTableUserPW') --> pw Debug: Config.pm ->Get('DatabaseUserTableUser') --> login Then the same line repeated about 40 times: Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1 Then it just segfaults again: [Fri Jun 11 19:13:12 2004] [notice] child pid 83975 exit signal Segmentation fault (11)
I hope this helps a bit,
Thanks for your help, at least I'm getting somewhere now :)
- --
- -jamie

On Friday, June 11, 2004 8:25 PM
Jamie L. Penman-Smithson
I've turned on debugging for all modules and still I can't see anything which would explain what's happening...
Me not either... let's see...
[Fri Jun 11 18:59:09 2004][Debug][Kernel::Language::Get][188] ->Get('Fri') Is not translated!!! [Fri Jun 11 18:59:09 2004][Debug][Kernel::Language::Get][188] ->Get('Jun') Is not translated!!!
No problem with these.
...and httpd-error.log seems to suggest that I'm missing some config parameters - this doesn't look as though it'd cause OTRS to segfault though?
Error: Config.pm No value for [...lalala...] in Config.pm found!
We're throwing errors here, in fact, these should be warnings, yes:
The last several lines are:
...all ok.
Then the same line repeated about 40 times:
Debug: Config.pm ->Get('DefaultCharset') --> iso-8859-1
Don't care.
Then it just segfaults again:
[Fri Jun 11 19:13:12 2004] [notice] child pid 83975 exit signal Segmentation fault (11)
Did you install Apache via a package or did you compile it yourself? I get the impression this is no real OTRS problem, but merely an Apache/Perl combi. What I could think of was disabling IPC session management ftm, although your system might well support it. Do so by using one these settings (default shown): $Self->{SessionModule} = 'Kernel::System::AuthSession::IPC'; # $Self->{SessionModule} = 'Kernel::System::AuthSession::DB'; # $Self->{SessionModule} = 'Kernel::System::AuthSession::FS'; Regards, Robert Kehl -- ((otrs.de)) :: OTRS GmbH :: Norsk-Data-Str. 1 :: 61352 Bad Homburg http://www.otrs.de/ :: Tel. +49 (0)6172 4832388

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 11 Jun 2004 21:30:18 +0200, Robert Kehl scribbled down:
Then it just segfaults again:
[Fri Jun 11 19:13:12 2004] [notice] child pid 83975 exit signal Segmentation fault (11)
Did you install Apache via a package or did you compile it yourself? I get the impression this is no real OTRS problem, but merely an Apache/Perl combi.
I installed apache and mod_perl from the fbsd ports tree, I've tried apache2 with mod_perl2 and apache1.3 with mod_perl1 - both exhibit the same problem, which is rather strange.
What I could think of was disabling IPC session management ftm, although your system might well support it. Do so by using one these settings (default shown):
$Self->{SessionModule} = 'Kernel::System::AuthSession::IPC'; # $Self->{SessionModule} = 'Kernel::System::AuthSession::DB'; # $Self->{SessionModule} = 'Kernel::System::AuthSession::FS';
It didn't make an ounce of difference - which seems to point to it being
something other than OTRS at play.
Thanks for all your help, now I've got to figure out where to go from here
*sigh*
Why can't life be simple? :p
- --
- -jamie

On Friday, June 11, 2004 10:59 PM
Jamie L. Penman-Smithson
Did you install Apache via a package or did you compile it yourself? I get the impression this is no real OTRS problem, but merely an Apache/Perl combi.
I installed apache and mod_perl from the fbsd ports tree, I've tried apache2 with mod_perl2 and apache1.3 with mod_perl1 - both exhibit the same problem, which is rather strange.
Recheck your httpd.conf files. Perhaps the mod_perls of the two servers bite each other? This is common on many installations. Try purging 1.3 if possible.
Why can't life be simple? :p
Because shit happens. *g* Robert Kehl -- ((otrs.de)) :: OTRS GmbH :: Norsk-Data-Str. 1 :: 61352 Bad Homburg http://www.otrs.de/ :: Tel. +49 (0)6172 4832388

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 11 Jun 2004 23:08:48 +0200, Robert Kehl scribbled down:
apache and mod_perl from the fbsd ports tree, I've tried apache2 with mod_perl2 and apache1.3 with mod_perl1 - both exhibit the same problem, which is rather strange.
Recheck your httpd.conf files.
Perhaps the mod_perls of the two servers bite each other? This is common on many installations. Try purging 1.3 if possible.
I've removed apache 1.3 and the old mod_perl, to no effect. The relevant
lines in my apache configuration are:
LoadModule perl_module libexec/apache2/mod_perl.so
PerlModule Apache2
<VirtualHost support.pinklemon.net:80>
ScriptAlias /otrs/ "/opt/otrs/bin/cgi-bin/"
Alias /otrs-web/ "/opt/otrs/var/httpd/htdocs/"
Perlrequire /opt/otrs/scripts/apache2-perl-startup.pl
PerlModule Apache::Reload
PerlInitHandler Apache::Reload
Why can't life be simple? :p
Because shit happens. *g*
Very true...
-j
--
-jamie

On Friday, June 11, 2004 11:59 PM
Jamie L. Penman-Smithson
I've removed apache 1.3 and the old mod_perl, to no effect.
What's your version of DBI? Try to use at least version 1.31, because of a known inconsistency with version <1.31 when using mod_perl. Also, your Apache is running in prefork mode. Although this is the most safest mode, it might not be the most efficient. It may well have something to do with your Apache dumping. Try to play with the other possible modes. Does it work _without_ modperl? Regards, Robert Kehl -- ((otrs.de)) :: OTRS GmbH :: Norsk-Data-Str. 1 :: 61352 Bad Homburg http://www.otrs.de/ :: Tel. +49 (0)6172 4832388

On Sat, 12 Jun 2004 10:26:24 +0200, Robert Kehl scribbled down:
On Friday, June 11, 2004 11:59 PM Jamie L. Penman-Smithson
wrote: I've removed apache 1.3 and the old mod_perl, to no effect.
What's your version of DBI? Try to use at least version 1.31, because of a known inconsistency with version <1.31 when using mod_perl.
I'm running version 1.42.
Also, your Apache is running in prefork mode. Although this is the most safest mode, it might not be the most efficient. It may well have something to do with your Apache dumping. Try to play with the other possible modes.
Does it work _without_ modperl?
Apache works fine without modperl, and even with modperl installed viewing
pages which don't use perl is not a problem.
I've set CoreDumpDirectory in apache but I don't think apache is at
fault. There are however, several files owned by the apache user in /tmp,
they're all zero size and seem to pop-up randomly.. Although I have a
feeling they're probably not related to the problem.
-rw------- 1 www wheel 0 Jun 13 18:37 apr65tu6A
-rw------- 1 www wheel 0 Jun 12 01:10 aprGV78B0
-rw------- 1 www wheel 0 Jun 13 18:37 aprg5PgdF
-rw------- 1 www wheel 0 Jun 12 01:10 aprkv6Zm9
[Sun Jun 13 18:39:59 2004] [notice] child pid 88216 exit signal Segmentation fault (11)
Child pid suggests that it's not apache, but perl or otrs instead?
--
-jamie

On Sunday, June 13, 2004 7:42 PM
Jamie L. Penman-Smithson
On Sat, 12 Jun 2004 10:26:24 +0200, Robert Kehl scribbled down:
Does it work _without_ modperl? I've set CoreDumpDirectory in apache but I don't think apache is at fault.
If so, you'd find some core dumps in the dir you specified. If it remains empty, it's indeed not Apache strinking the wings.
There are however, several files owned by the apache user in /tmp,
I've no idea about these.
[Sun Jun 13 18:39:59 2004] [notice] child pid 88216 exit signal Segmentation fault (11) Child pid suggests that it's not apache, but perl or otrs instead?
OTRS cannot die, as it's a Perl app. Therefore, it might be Perl that's exiting. Check that, you need to know exactly which process it is. When does the issue happen exactly. I.e. which action do you perform that invokes it? Regards, Robert Kehl -- ((otrs.de)) :: OTRS GmbH :: Norsk-Data-Str. 1 :: 61352 Bad Homburg http://www.otrs.de/ :: Tel. +49 (0)6172 4832388
participants (2)
-
Jamie L. Penman-Smithson
-
Robert Kehl