Re: [otrs] Problem upgrading to 3.2.8 on Windows (cause?: old HTTPD binaries in latest installer?)

Today we managed to orchestrate a load test for ~45 minutes with 20 agents on our QA instance running otrs-3.2.3-win-installer-2.4.5. Tests passed without any resurfacing of the dreaded IPHLPAPI.DLL crash. Also noteworthy, tests started when the QA env was in its 7th day of uptime, during which the error was nowhere to be seen. In conclusion, we'll upgrade to this 3.2 version. In my opinion, changes to the Strawberry, Apache or MySQL versions packaged with the Windows installer should be introduced between revisions ONLY if it's dramatically important to do so, in which case they should at least be mentioned somewhere. On Tue, Jun 25, 2013 at 12:22 PM, Bogdan Iosif wrote:

Entering the 5th day of stability on QA with otrs-3.2.3-win-installer-2.4.5 (4 days of uptime, light testing, no errors). I'll post a new update after we manage to do some manual load testing.

I would say my suspicion was confirmed. The problem was introduced with the updated Strawberry from otrs-3.2.3-win-installer-2.4.6 or otrs-3.2.4-win-installer-2.4.7. Maybe it only manifests itself on a configuration like mine (Windows 2008 R2 Enterprise x64, with SP1, fully updated via Windows Update)

For now, I need to keep the QA environment stable and can't play around with various other versions. However, if all current tests are OK and we deploy 3.2.3 on prod, then starting next week I'll be able to free up QA and resume 3.2.8 testing. While I'm aware I'll lose a lot of fixes introduced since 3.2.3, I think first upgrading to a stable 3.2.3 allows me to move from 3.1 to 3.2 in a safer manner (if problems persist when I'll move from 3.2.3 to 3.2.8 then at least I'll be able to rollback to 3.2.3 without losing db-level data because there are no db changes between these two revisions).

I would like to deploy again otrs-3.2.8-win-installer-2.4.8, because it produces the error immediately, but instead of manually fiddling with various Win32.pm versions which would create an environment nobody else can reliably reproduce I would like to enable the most extensive debug mode available in OTRS coupled with some process crash monitoring tools. In this way, I'll hopefully be able to produce an HTTPD memory dump and see exactly which Win32 method call is causing the crash.

If you would assist me on tuning OTRS's debug mode, I'll take care of the rest.

On Mon, Jun 24, 2013 at 11:19 AM, Michiel Beijen

...

wrote:

...

BTW if you want to test - here is Win32 version 0.44. http://users.otrs.com/~mb/perl-Win32-0.44.zip ==> to install it, simply unzip the contents in ..\StrawberryPerl\perl\site\lib

-- Mike

On Sat, Jun 22, 2013 at 2:22 PM, Michiel Beijen

...

wrote:

...

Hi Bogdan,

First of all, thanks for all the efforts you're undertaking here!

Second, I think your conclusion of blaming Win32.pm might be a red herring. The reason for including the newer version is quite simple - one of the functions we use it for is to determine the actual version of Windows it is running on. This is used in our Support Module. I added support for detecting Windows 2012 Server and Windows 8, this was introduced in version 0.46. The full list of changes is found on CPAN; you can also use metacpan to make diffs between releases. Unfortunately there are lots of whitespace changes in one of the releases so not all is fully trackable, but all the changes seem not to have much impact. https://metacpan.org/release/JDB/Win32-0.47

If you want to install a different version of Win32 you can install it like this:

cpan JDB/Win32-0.43.tar.gz

Please note that this only works if you have installed OTRS in a path without spaces, i.e. not in C:\Program Files, but in d:\Helpdesk would work.

Again, if the Win32 change would be the culprit, this would be super but I have my doubts.

The Version.pm that you found introduced in OTRS was because we started using Encode::Locale to fix encoding issues for command line utilities, and they require Version.pm which is core, but only in perl 5.10 and later.

For the 2.4.9 version you tested I basically took the Perl and Apache versions of an old, stable installer and put them into the new one. BTW other people stated that for *them* the 2.4.9 actually works.

Was one of the versions of OTRS on Windows for you really stable? In my experience the mod_perl based ones are generally stable, but sometimes the web server restarts. This could be as simple as the worker restarting (see the setting in the apache config). The last version of the windows installer also stops, and you'd have to start it manually, of course this is NOT desired and I'd like to learn why this is and what we could do about it. If there was a version that did work for you, can you use THAT one, and replace the OTRS with 3.2.8 and see if THAT is stable? It *could* be we simply introduced breakage by accident in the OTRS code base.

The best results on WIndows I found are when you install ActiveState perl (x86) which comes with PerlEx, and you use Microsoft IIS as the web server. This is also what the 'new' beta installer advertises.

Looking forward reading your answers,

-- Mike

On Thu, Jun 20, 2013 at 6:54 PM, Bogdan Iosif wrote:

...

The problem still exists in a deployment made with otrs-3.2.8-win-installer-2.4.9. The HTTPD process managed to survive a little longer in this version but it's still unstable and crashes with the same pattern.

I'll first jump to conclusions. I think the problem's root cause was incorrectly identified so far and it's likely caused by changes in the Perl distribution deployed by the OTRS Windows installer. To be more exact, I think an upgraded Win32.pm deployed since otrs-3.2.3-win-installer-2.4.6 is close to the root cause.

I downloaded all the 3.2 installation packages between otrs-3.2.3-win-installer-2.4.5 and the latest otrs-3.2.8-win-installer-2.4.9 provided by Michiel. Then I proceeded to unpack them and to identify differences. Below are the results.

I chose to start with otrs-3.2.3-win-installer-2.4.5 because my current prod env was deployed with otrs-3.1.10-win-installer-2.4.5 and I made the common sense assumption that the installer logic hasn't changed between these two packages.

In the notes found below all dirs are written with a trailing \ and all root dirs are actually relative to $_OUTDIR\ dir found in their otrs-#.#.#-win-installer-#.#.#.exe package.

Additional comments after these notes.

~ otrs-3.2.3-win-installer-2.4.5 VERSUS otrs-3.1.10-win-installer-2.4.5 - \OTRS\ = Lots of changes # Expected as a normal part of an OTRS version upgrade - \StrawberryPerl\ = Some changes # Expected between two major versions

~ otrs-3.2.3-win-installer-2.4.6 VERSUS previous - \Scripts\ = Changed ConfigureOTRS.pl # Added single quotes around LogModule and LogModule::LogFile written in Config.pm - \StrawberryPerl\ = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?!

~ otrs-3.2.4-win-installer-2.4.7 VERSUS previous - \OTRS\ = Lots of changes # Expected as a normal part of an OTRS version upgrade - \Scripts\ = ConfigureApache.pl changed to add LoadModule directives for mod_deflate.so and mod_headers.so # Ignorable? Is this safe to occur with an OTRS minor version upgrade? = Lots of other changes # Ignorable. Updates to copyright years, licensing details, file versions, etc. - \StrawberryPerl\ = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?! Many files seem to have just been relocated.

~ otrs-3.2.5-win-installer-2.4.7 VERSUS previous - \OTRS\ = Lots of changes # Expected as a normal part of an OTRS version upgrade - \StrawberryPerl\ = perl\site\lib\version.pm moved here from \OTRS\Kernel\cpan-lib\ # UNEXPECTED with an OTRS minor version upgrade?!

~ otrs-3.2.6-win-installer-2.4.7 VERSUS previous - \OTRS\ = Lots of changes # Expected as a normal part of an OTRS version upgrade

~ otrs-3.2.7-win-installer-2.4.7 VERSUS previous - \OTRS\ = Lots of changes # Expected as a normal part of an OTRS version upgrade

~ otrs-3.2.8-win-installer-2.4.8 VERSUS previous - \Apache\ = Replaced many bin\*.dll (including libapr-1.dll) with older versions from 2009 (versions from otrs:3.2.7 were from 2012) # UNEXPECTED with an OTRS minor version upgrade?! = Added *.pdb files for the replaced *.dll # UNEXPECTED with an OTRS minor version upgrade?! Ignorable? PDBs are presumably ignored by Apache. = Replaced many bin\iconv\*.so with older versions from 2008 (versions from otrs:3.2.7 were from 2012) # UNEXPECTED with an OTRS minor version upgrade?! = Added *.pdb files for the replaced *.so. # UNEXPECTED with an OTRS minor version upgrade?! Ignorable? PDBs are presumably ignored by Apache. - \OTRS\ = Lots of changes # Expected as a normal part of an OTRS version upgrade

~ otrs-3.2.8-win-installer-2.4.9 VERSUS previous - \Apache\ = Lots of changes # Rolled back to the version included up to otrs:3.2.7 except for conf\httpd.conf which has some ignorable changes to default config values that will later be set anyway by the installer - \StrawberryPerl\ = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?!

~ otrs-3.2.8-win-installer-2.4.9 VERSUS otrs-3.2.7-win-installer-2.4.7 - \Apache\conf\ = Changed httpd.conf # Ignorable. Different default config values that will later be set anyway by the installer. - \OTRS\ = Lots of changes # Expected as a normal part of an OTRS version upgrade - \StrawberryPerl\ = Lots of changes # UNEXPECTED with an OTRS minor version upgrade?!

otrs-3.2.8-win-installer-2.4.9 introduced some trivial changes in mysql and cronw that were ignored and not listed

In this forum thread http://forums.otterhub.org/viewtopic.php?f=63&t=20288 a user reports the problem occurring with otrs-3.2.6-win-installer-2.4.7 and otrs-3.2.5-win-installer-2.4.7. Looking at the differences listed above, it makes sense. These two package versions contain only differences in OTRS sources. HOWEVER, the old HTTPD DLLs were not yet deployed by otrs-3.2.6-win-installer-2.4.7 so they can't possibly be the root cause for the problem.

My event log crash entries mention something about IPHLPAPI.DLL. Googling this dll and Perl I found lots of error reports, most having to do with using 64-bit Perl, which is not the case for me. However, then I tought that maybe something changed in the chain of scripts used by OTRS to call this DLL. IPHelper.pm hasn't changed in a long time but one of its dependencies, Win32.pm, made a rather large version jump from 0.44 (~2011) to 0.47 (~2013) in the Perl dist. deployed by otrs-3.2.3-win-installer-2.4.6.

My current bet for the root cause of this problem is the aforementioned upgrade for Win32.pm. I'll install otrs-3.2.3-win-installer-2.4.5 and test how right I am about this.

I'm hoping for some feedback from someone who knows about the reason behind the Win32.pm upgrade and maybe also knows how to test if this is actually the root cause of the problem.

/bogdan

On Wed, Jun 19, 2013 at 10:57 PM, Bogdan Iosif wrote:

...

I'll set it up and give feedback in about 6 - 10 hours. Thanks!

My trouble is... I think I just missed a maintenance window for this weekend to perform the upgrade to 3.2.8 because this situation looks like a stability problem with my configuration that I simply haven't planned to protect against.

To tell the truth, I feel lucky it happened within the first few clicks because if it would've appeared on QA after a couple hours of light testing then I would've seen it in production within about 30 minutes to an hour and I would've had no way to rollback to 3.1.10. That would've been a nightmare.

If I see that 3.2.8 doesn't crash immediately anymore, is there anything you can think of that I can try to further ensure the stability of the configuration deployed by this latest installer?

On Wed, Jun 19, 2013 at 10:07 PM, Michiel Beijen < michiel.beijen@gmail.com> wrote:

...

Hi Bogdan,

We switched to a newer version of mod_perl which was compiled by someone from apache; aparently with iconv and libapr versions that are causing all sorts of trouble. Very annoying.

I created a 3.2.8 version based on the 'old' apache, mod_perl and strawberry, please let me know if it helps!

http://users.otrs.com/~mb/otrs-3.2.8-win-installer-2.4.9.exe --

On Wed, Jun 19, 2013 at 3:37 PM, Bogdan Iosif wrote:

> Hi list, > > I'm trying to upgrade to 3.2.8, from 3.1.10, on Win 2008 R2 with SP1 > and I'm running into an ugly problem that I think is related to the Apache > config deployed by OTRS's Windows installer 2.4.8. I used this 3.2.8 > package because I made the common sense assumption that its installer is > more stable than the latest beta for OTRS's Windows installer 3.0.0. > > First, I performed a clean install (in a VM where Windows itself was > freshly installed) using otrs-3.2.8-win-installer-2.4.8.exe. When it came > time to run the web installer I instead switched to following instructions > for the upgrade from here: > https://github.com/OTRS/otrs/blob/rel-3_2_8/UPGRADING.md > > =Symptoms= > > Everything went OK with the upgrade up to, and through, step "12. > Restart your services". > > When I reached step 13 and needed to start using OTRS's admin > interface, I discovered the application is unusable because Apache's > process crashes very suddenly after a few HTTP requests and OTRS is > effectively shutdown afterwards. If I attempt to manually restart the > Apache service then it crashes again very soon, following the same pattern. > The Event Log contains some cryptic messages like: > > ==System Log== > > The Apache2.2 service terminated unexpectedly. > > ==Application Log== > > Faulting application name: httpd.exe, version: 2.2.22.0, time stamp: > 0x4f242d7a > Faulting module name: unknown, version: 0.0.0.0, time stamp: > 0x00000000 > Exception code: 0xc0000005 > Fault offset: 0x753a34c1 > Faulting process id: 0x458 > Faulting application start time: 0x01ce6cdea9707025 > Faulting application path: C:\Program Files > (x86)\OTRS\Apache\bin\httpd.exe > Faulting module path: unknown > Report Id: 3c668618-d8d2-11e2-b6da-000c29f4581a > > Faulting application name: httpd.exe, version: 2.2.22.0, time stamp: > 0x4f242d7a > Faulting module name: IPHLPAPI.DLL_unloaded, version: 0.0.0.0, time > stamp: 0x4ce7b859 > Exception code: 0xc0000005 > Fault offset: 0x753a34c1 > Faulting process id: 0x6bc > Faulting application start time: 0x01ce6cdeb6fa43b4 > Faulting application path: C:\Program Files > (x86)\OTRS\Apache\bin\httpd.exe > Faulting module path: IPHLPAPI.DLL > Report Id: 3bfdbf74-d8d2-11e2-b6da-000c29f4581a > > =My Investigation= > > The following recent post (in German) describes almost the same > exact problem that I have: > http://stefan.pachlina.net/index.php?option=com_content&view=article&id=317:... > > Reading through Google Translate, I saw the author pinpointing the > root cause to a faulty version of \OTRS\apache\bin\libapr-1.dll. He > said the bad version is 1.4.5.0 and recommends instead either 1.3.6.0 or > 1.4.6.0. Note that I currently have 1.4.5.0 in production and haven't ran > into any problems. Also, I have no idea how the post author identified that > DLL to be the problem. > > The output from "httpd -v" is identical between my prod (3.1.10) and > test (3.2.8) envs but after doing a folder comparison I see a lot of > differences in files from apache\bin. First, there are a lot of *.DLL and > *.SO files that are different and the versions deployed with 3.2.8 seem a > lot older (from 2009) than those deployed with 3.1.10 (from 2012). Here is > a report (copy/paste in notepad to see it clearly): > > Name Version Size Modified > Name Version Size Modified > > ------------------------------------------------------------------------------------------------------------------------------- > bin 9`108`428 19-Jun-2013 15:52:38 > bin 28`155`388 19-Jun-2013 15:50:52 > +libaprutil-1.dll 1.4.1.0 192`604 28-Jan-2012 12:15:46 >> > +libaprutil-1.dll 1.3.8.0 188`496 06-Jul-2009 23:21:36 > +libapr-1.dll 1.4.5.0 139`347 28-Jan-2012 12:11:16 >> > +libapr-1.dll 1.3.6.0 135`239 06-Jul-2009 23:21:20 > +libapriconv-1.dll 1.2.1.0 36`958 28-Jan-2012 12:11:24 >> > +libapriconv-1.dll 1.2.1.0 36`946 06-Jul-2009 23:21:24 > +apr_dbd_oracle-1.dll 1.4.1.0 32`868 28-Jan-2012 12:15:54 >> > +apr_dbd_oracle-1.dll 1.3.8.0 32`856 06-Jul-2009 23:21:38 > +apr_dbd_sqlite3-1.dll 1.4.1.0 28`773 28-Jan-2012 12:15:52 >> > +apr_dbd_sqlite3-1.dll 1.3.8.0 28`761 06-Jul-2009 23:21:38 > +apr_dbd_pgsql-1.dll 1.4.1.0 28`771 28-Jan-2012 12:15:56 >> > +apr_dbd_pgsql-1.dll 1.3.8.0 28`759 06-Jul-2009 23:21:38 > +apr_dbd_mysql-1.dll 1.4.1.0 28`771 28-Jan-2012 12:15:58 >> > +apr_dbd_mysql-1.dll 1.3.8.0 28`759 06-Jul-2009 23:21:38 > +apr_dbd_odbc-1.dll 28`770 28-Jan-2012 12:15:50 >> > +apr_dbd_odbc-1.dll 28`758 06-Jul-2009 23:21:38 > \apr_ldap-1.dll 1.4.1.0 24`671 28-Jan-2012 12:15:50 >> > \apr_ldap-1.dll 1.3.8.0 24`659 06-Jul-2009 23:21:36 > > ------------------------------------------------------------------------------------------------------------------------------- > > There are more differences, amongst which the most notable are a lot > of *.SO files in \apache\bin\iconv that are even older (from 2008). Last > but not least, there are a lot of *.PDB files sprinkled through \bin > subfolders. > > =Bottom Line= > > I don't know what to make out of this difference in DLL versions. It > seems the installer's author started to use older versions on purpose but I > don't know why. > > Can someone make sense of this problem or at least provide me with > some info about a 3.2.x release whose Windows installer works on Windows > 2008 R2? > > /bogdan > > --------------------------------------------------------------------- > OTRS mailing list: otrs - Webpage: http://otrs.org/ > Archive: http://lists.otrs.org/pipermail/otrs > To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs >

--------------------------------------------------------------------- OTRS mailing list: otrs - Webpage: http://otrs.org/ Archive: http://lists.otrs.org/pipermail/otrs To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs

--------------------------------------------------------------------- OTRS mailing list: otrs - Webpage: http://otrs.org/ Archive: http://lists.otrs.org/pipermail/otrs To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs