The problem still exists in a deployment made with otrs-3.2.8-win-installer-2.4.9. The HTTPD process managed to survive a little longer in this version but it's still unstable and crashes with the same pattern.

I'll first jump to conclusions. I think the problem's root cause was incorrectly identified so far and it's likely caused by changes in the Perl distribution deployed by the OTRS Windows installer. To be more exact, I think an upgraded Win32.pm deployed since otrs-3.2.3-win-installer-2.4.6 is close to the root cause.

I downloaded all the 3.2 installation packages between otrs-3.2.3-win-installer-2.4.5 and the latest otrs-3.2.8-win-installer-2.4.9 provided by Michiel. Then I proceeded to unpack them and to identify differences. Below are the results.

I chose to start with otrs-3.2.3-win-installer-2.4.5 because my current prod env was deployed with otrs-3.1.10-win-installer-2.4.5 and I made the common sense assumption that the installer logic hasn't changed between these two packages.

In the notes found below all dirs are written with a trailing \ and all root dirs are actually relative to $_OUTDIR\ dir found in their otrs-#.#.#-win-installer-#.#.#.exe package.

Additional comments after these notes.

~ otrs-3.2.3-win-installer-2.4.5 VERSUS otrs-3.1.10-win-installer-2.4.5
    - \OTRS\            = Lots of changes # Expected as a normal part of an OTRS version upgrade
    - \StrawberryPerl\  = Some changes # Expected between two major versions

~ otrs-3.2.3-win-installer-2.4.6 VERSUS previous
    - \Scripts\         = Changed ConfigureOTRS.pl # Added single quotes around LogModule and LogModule::LogFile written in Config.pm
    - \StrawberryPerl\  = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?!

~ otrs-3.2.4-win-installer-2.4.7 VERSUS previous
    - \OTRS\            = Lots of changes # Expected as a normal part of an OTRS version upgrade
    - \Scripts\         = ConfigureApache.pl changed to add LoadModule directives for mod_deflate.so and mod_headers.so # Ignorable? Is this safe to occur with an OTRS minor version upgrade?
                        = Lots of other changes # Ignorable. Updates to copyright years, licensing details, file versions, etc.
    - \StrawberryPerl\  = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?! Many files seem to have just been relocated.

~ otrs-3.2.5-win-installer-2.4.7 VERSUS previous
    - \OTRS\            = Lots of changes # Expected as a normal part of an OTRS version upgrade
    - \StrawberryPerl\  = perl\site\lib\version.pm moved here from \OTRS\Kernel\cpan-lib\ # UNEXPECTED with an OTRS minor version upgrade?!

~ otrs-3.2.6-win-installer-2.4.7 VERSUS previous
    - \OTRS\            = Lots of changes # Expected as a normal part of an OTRS version upgrade

~ otrs-3.2.7-win-installer-2.4.7 VERSUS previous
    - \OTRS\            = Lots of changes # Expected as a normal part of an OTRS version upgrade

~ otrs-3.2.8-win-installer-2.4.8 VERSUS previous
    - \Apache\          = Replaced many bin\*.dll (including libapr-1.dll) with older versions from 2009 (versions from otrs:3.2.7 were from 2012) # UNEXPECTED with an OTRS minor version upgrade?!
                        = Added *.pdb files for the replaced *.dll # UNEXPECTED with an OTRS minor version upgrade?! Ignorable? PDBs are presumably ignored by Apache.
                        = Replaced many bin\iconv\*.so with older versions from 2008 (versions from otrs:3.2.7 were from 2012) # UNEXPECTED with an OTRS minor version upgrade?!
                        = Added *.pdb files for the replaced *.so. # UNEXPECTED with an OTRS minor version upgrade?! Ignorable? PDBs are presumably ignored by Apache.
    - \OTRS\            = Lots of changes # Expected as a normal part of an OTRS version upgrade

~ otrs-3.2.8-win-installer-2.4.9 VERSUS previous
    - \Apache\          = Lots of changes # Rolled back to the version included up to otrs:3.2.7 except for conf\httpd.conf which has some ignorable changes to default config values that will later be set anyway by the installer
    - \StrawberryPerl\  = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?!

~ otrs-3.2.8-win-installer-2.4.9 VERSUS otrs-3.2.7-win-installer-2.4.7
    - \Apache\conf\     = Changed httpd.conf # Ignorable. Different default config values that will later be set anyway by the installer.
    - \OTRS\            = Lots of changes # Expected as a normal part of an OTRS version upgrade
    - \StrawberryPerl\  = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?!


otrs-3.2.8-win-installer-2.4.9 introduced some trivial changes in mysql and cronw that were ignored and not listed

In this forum thread http://forums.otterhub.org/viewtopic.php?f=63&t=20288 a user reports the problem occurring with otrs-3.2.6-win-installer-2.4.7 and otrs-3.2.5-win-installer-2.4.7. Looking at the differences listed above, it makes sense. These two package versions contain only differences in OTRS sources. HOWEVER, the old HTTPD DLLs were not yet deployed by otrs-3.2.6-win-installer-2.4.7 so they can't possibly be the root cause for the problem.

My event log crash entries mention something about IPHLPAPI.DLL. Googling this dll and Perl I found lots of error reports, most having to do with using 64-bit Perl, which is not the case for me. However, then I tought that maybe something changed in the chain of scripts used by OTRS to call this DLL. IPHelper.pm hasn't changed in a long time but one of its dependencies, Win32.pm, made a rather large version jump from 0.44 (~2011) to 0.47 (~2013) in the Perl dist. deployed by otrs-3.2.3-win-installer-2.4.6.

My current bet for the root cause of this problem is the aforementioned upgrade for Win32.pm. I'll install otrs-3.2.3-win-installer-2.4.5 and test how right I am about this.

I'm hoping for some feedback from someone who knows about the reason behind the Win32.pm upgrade and maybe also knows how to test if this is actually the root cause of the problem.

/bogdan



On Wed, Jun 19, 2013 at 10:57 PM, Bogdan Iosif <bogdan.iosif@gmail.com> wrote:
I'll set it up and give feedback in about 6 - 10 hours. Thanks!

My trouble is... I think I just missed a maintenance window for this weekend to perform the upgrade to 3.2.8 because this situation looks like a stability problem with my configuration that I simply haven't planned to protect against.

To tell the truth, I feel lucky it happened within the first few clicks because if it would've appeared on QA after a couple hours of light testing then I would've seen it in production within about 30 minutes to an hour and I would've had no way to rollback to 3.1.10. That would've been a nightmare.

If I see that 3.2.8 doesn't crash immediately anymore, is there anything you can think of that I can try to further ensure the stability of  the configuration deployed by this latest installer?



On Wed, Jun 19, 2013 at 10:07 PM, Michiel Beijen <michiel.beijen@gmail.com> wrote:
Hi Bogdan,

We switched to a newer version of mod_perl which was compiled by someone from apache; aparently with iconv and libapr versions that are causing all sorts of trouble. Very annoying.

I created a 3.2.8 version based on the 'old'  apache, mod_perl and strawberry, please let me know if it helps!

http://users.otrs.com/~mb/otrs-3.2.8-win-installer-2.4.9.exe
--


On Wed, Jun 19, 2013 at 3:37 PM, Bogdan Iosif <bogdan.iosif@gmail.com> wrote:
Hi list,

I'm trying to upgrade to 3.2.8, from 3.1.10, on Win 2008 R2 with SP1 and I'm running into an ugly problem that I think is related to the Apache config deployed by OTRS's Windows installer 2.4.8. I used this 3.2.8 package because I made the common sense assumption that its installer is more stable than the latest beta for OTRS's Windows installer 3.0.0.

First, I performed a clean install (in a VM where Windows itself was freshly installed) using otrs-3.2.8-win-installer-2.4.8.exe. When it came time to run the web installer I instead switched to following instructions for the upgrade from here: https://github.com/OTRS/otrs/blob/rel-3_2_8/UPGRADING.md

=Symptoms=

Everything went OK with the upgrade up to, and through, step "12. Restart your services".

When I reached step 13 and needed to start using OTRS's admin interface, I discovered the application is unusable because Apache's process crashes very suddenly after a few HTTP requests and OTRS is effectively shutdown afterwards. If I attempt to manually restart the Apache service then it crashes again very soon, following the same pattern. The Event Log contains some cryptic messages like:

==System Log==

The Apache2.2 service terminated unexpectedly.

==Application Log==

Faulting application name: httpd.exe, version: 2.2.22.0, time stamp: 0x4f242d7a
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x753a34c1
Faulting process id: 0x458
Faulting application start time: 0x01ce6cdea9707025
Faulting application path: C:\Program Files (x86)\OTRS\Apache\bin\httpd.exe
Faulting module path: unknown
Report Id: 3c668618-d8d2-11e2-b6da-000c29f4581a

Faulting application name: httpd.exe, version: 2.2.22.0, time stamp: 0x4f242d7a
Faulting module name: IPHLPAPI.DLL_unloaded, version: 0.0.0.0, time stamp: 0x4ce7b859
Exception code: 0xc0000005
Fault offset: 0x753a34c1
Faulting process id: 0x6bc
Faulting application start time: 0x01ce6cdeb6fa43b4
Faulting application path: C:\Program Files (x86)\OTRS\Apache\bin\httpd.exe
Faulting module path: IPHLPAPI.DLL
Report Id: 3bfdbf74-d8d2-11e2-b6da-000c29f4581a

=My Investigation=

Reading through Google Translate, I saw the author pinpointing the root cause to a faulty version of \OTRS\apache\bin\libapr-1.dll. He said the bad version is 1.4.5.0 and recommends instead either 1.3.6.0 or 1.4.6.0. Note that I currently have 1.4.5.0 in production and haven't ran into any problems. Also, I have no idea how the post author identified that DLL to be the problem.

The output from "httpd -v" is identical between my prod (3.1.10) and test (3.2.8) envs but after doing a folder comparison I see a lot of differences in files from apache\bin. First, there are a lot of *.DLL and *.SO files that are different and the versions deployed with 3.2.8 seem a lot older (from 2009) than those deployed with 3.1.10 (from 2012). Here is a report (copy/paste in notepad to see it clearly):

Name                   Version Size      Modified                Name                   Version Size       Modified
-------------------------------------------------------------------------------------------------------------------------------
bin                            9`108`428 19-Jun-2013 15:52:38    bin                            28`155`388 19-Jun-2013 15:50:52
+libaprutil-1.dll      1.4.1.0 192`604   28-Jan-2012 12:15:46 >> +libaprutil-1.dll      1.3.8.0 188`496    06-Jul-2009 23:21:36
+libapr-1.dll          1.4.5.0 139`347   28-Jan-2012 12:11:16 >> +libapr-1.dll          1.3.6.0 135`239    06-Jul-2009 23:21:20
+libapriconv-1.dll     1.2.1.0 36`958    28-Jan-2012 12:11:24 >> +libapriconv-1.dll     1.2.1.0 36`946     06-Jul-2009 23:21:24
+apr_dbd_oracle-1.dll  1.4.1.0 32`868    28-Jan-2012 12:15:54 >> +apr_dbd_oracle-1.dll  1.3.8.0 32`856     06-Jul-2009 23:21:38
+apr_dbd_sqlite3-1.dll 1.4.1.0 28`773    28-Jan-2012 12:15:52 >> +apr_dbd_sqlite3-1.dll 1.3.8.0 28`761     06-Jul-2009 23:21:38
+apr_dbd_pgsql-1.dll   1.4.1.0 28`771    28-Jan-2012 12:15:56 >> +apr_dbd_pgsql-1.dll   1.3.8.0 28`759     06-Jul-2009 23:21:38
+apr_dbd_mysql-1.dll   1.4.1.0 28`771    28-Jan-2012 12:15:58 >> +apr_dbd_mysql-1.dll   1.3.8.0 28`759     06-Jul-2009 23:21:38
+apr_dbd_odbc-1.dll            28`770    28-Jan-2012 12:15:50 >> +apr_dbd_odbc-1.dll            28`758     06-Jul-2009 23:21:38
\apr_ldap-1.dll        1.4.1.0 24`671    28-Jan-2012 12:15:50 >> \apr_ldap-1.dll        1.3.8.0 24`659     06-Jul-2009 23:21:36
-------------------------------------------------------------------------------------------------------------------------------

There are more differences, amongst which the most notable are a lot of *.SO files in \apache\bin\iconv that are even older (from 2008). Last but not least, there are a lot of *.PDB files sprinkled through \bin subfolders.

=Bottom Line=

I don't know what to make out of this difference in DLL versions. It seems the installer's author started to use older versions on purpose but I don't know why.

Can someone make sense of this problem or at least provide me with some info about a 3.2.x release whose Windows installer works on Windows 2008 R2?

/bogdan

---------------------------------------------------------------------
OTRS mailing list: otrs - Webpage: http://otrs.org/
Archive: http://lists.otrs.org/pipermail/otrs
To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs


---------------------------------------------------------------------
OTRS mailing list: otrs - Webpage: http://otrs.org/
Archive: http://lists.otrs.org/pipermail/otrs
To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs