I currently have an OTRS (4.7) install on Centos 6.6. We are using a remote MySQL database (remote as in not on the same machine as the OTRS installation but in the same network vlan). Is this a recommended or supported setup? We are seeing slowdowns of up to 15 minutes on the OTRS system occasionally. Thank You Dennis
Am 17.04.2015 um 18:16 schrieb Ritchie, Dennis:
I currently have an OTRS (4.7) install on Centos 6.6.
We are using a remote MySQL database (remote as in not on the same machine as the OTRS installation but in the same network vlan).
Is this a recommended or supported setup?
We are seeing slowdowns of up to 15 minutes on the OTRS system occasionally.
Thank You
Dennis
Hi Dennis, Using a remote MySQL database is usually no Problem - we do it all the time. 15 Minutes (??) For what? Getting the next Page or getting next Ticktes? Do you have any ErrorMessages in the orts-log? Regards, Fritz -- ------------------------------------- Friedrich Koelbel http://www.graz4u.at -------------------------------------
From: otrs-bounces@otrs.org [mailto:otrs-bounces@otrs.org] On Behalf Of Friedrich Kölbel Sent: Friday, April 17, 2015 1:10 PM To: User questions and discussions about OTRS. Subject: Re: [otrs] question Am 17.04.2015 um 18:16 schrieb Ritchie, Dennis: I currently have an OTRS (4.7) install on Centos 6.6. We are using a remote MySQL database (remote as in not on the same machine as the OTRS installation but in the same network vlan). Is this a recommended or supported setup? We are seeing slowdowns of up to 15 minutes on the OTRS system occasionally. Thank You Dennis Hi Dennis, Using a remote MySQL database is usually no Problem - we do it all the time. 15 Minutes (??) For what? Getting the next Page or getting next Ticktes? Do you have any ErrorMessages in the orts-log? Regards, Fritz -- ------------------------------------- Friedrich Koelbel http://www.graz4u.at ------------------------------------- Thanks for the response, I answered the for what in a different message but for continuity: The 15 minutes can be to get a log on screen, to switch screens inside of the application when running, to open or close a ticket. Basically anything at all with regards to using the system. I know that this can't be the way this ordinarily runs, so I assume that something that we have setup is doing it, but for the life of me I don't know what/where. That's why I asked about the remote MySQL. I've examined the /opt/otrs/var/log directory and there is only the TicketCounter.log with any data, all other logs (Scheduler*.log) are empty. The system /var/log/httpd/access_log shows that the system gets a request and then just waits: 10.x.x.x - - [17/Apr/2015:10:05:09 -0500] "GET / HTTP/1.1" 302 26 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.18 Safari/537.36" The system /var/log/httpd/error_log has nothing The System /var/messages has nothing in the way of errors, it shows a log - on then nothing until the system is responsive again. Thank You Dennis
We are using a remote MySQL database (remote as in not on the same machine as the OTRS installation but in the same network vlan). Is this a recommended or supported setup? We are seeing slowdowns of up to 15 minutes on the OTRS system occasionally. Yes, that's a perfectly fine setup, but.... it's usually a good idea to not share the same database server instance with other workload unless you really understand how the different workloads use the database server and you can really measure how the different workloads behave. Naively-constructed queries from other workloads to other tables in a shared database server can make OTRS unhappy if the other workload uses long-lasting locks on system catalog resources, or other shared resources in the database server. Very few developers play nice in shared environments these days - designing good code is hard; throwing more hardware at the problem is lots easier. Recent versions of OTRS have been much, much better about playing nice - kudos to the OTRS devs. When you say 15 minutes, is that wait time for searches or adding new records or ??? If that's adding new records, there's definitely something not cool with the database server or application server configuration. Do you have a performance monitor on the database server? If so, what does it say? Is your database server set for table or record-level locking? How many simultaneous sessions? Average ticket volume? Need more data.
The 15 minutes can be to get a logon screen. Or if already logged in to switch screens. Thank You Dennis From: otrs-bounces@otrs.org [mailto:otrs-bounces@otrs.org] On Behalf Of David Boyes Sent: Friday, April 17, 2015 1:16 PM To: User questions and discussions about OTRS. Subject: Re: [otrs] question We are using a remote MySQL database (remote as in not on the same machine as the OTRS installation but in the same network vlan). Is this a recommended or supported setup? We are seeing slowdowns of up to 15 minutes on the OTRS system occasionally. Yes, that's a perfectly fine setup, but.... it's usually a good idea to not share the same database server instance with other workload unless you really understand how the different workloads use the database server and you can really measure how the different workloads behave. Naively-constructed queries from other workloads to other tables in a shared database server can make OTRS unhappy if the other workload uses long-lasting locks on system catalog resources, or other shared resources in the database server. Very few developers play nice in shared environments these days - designing good code is hard; throwing more hardware at the problem is lots easier. Recent versions of OTRS have been much, much better about playing nice - kudos to the OTRS devs. When you say 15 minutes, is that wait time for searches or adding new records or ??? If that's adding new records, there's definitely something not cool with the database server or application server configuration. Do you have a performance monitor on the database server? If so, what does it say? Is your database server set for table or record-level locking? How many simultaneous sessions? Average ticket volume? Need more data.
The 15 minutes can be to get a logon screen. Or if already logged in to switch screens. That sounds more like a resource constraint problem on the server running the OTRS code than a database problem - OTRS is a relatively large application, and can be fairly demanding on application server resources. Try monitoring CPU, swap and memory utilization at 5 minute intervals on the OTRS server for a few days and see if there's any correlation to the problem periods - if the OTRS server is thrashing, the database response is irrelevant. Eliminate that possibility first, and then start looking at database performance.
<quote> That sounds more like a resource constraint problem on the server </quote> I agree but see below (I've munged some of the information): Running Fine: top - 11:25:54 up 33 days, 16:05, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 256 total, 1 running, 255 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3872832k total, 2476000k used, 1396832k free, 217676k buffers Swap: 2064380k total, 11676k used, 2052704k free, 1092156k cached Not responding: top - 13:42:33 up 33 days, 18:22, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 256 total, 1 running, 255 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3872832k total, 2564764k used, 1308068k free, 218068k buffers Swap: 2064380k total, 11676k used, 2052704k free, 1103772k cached System Log (from the Admin Page of the website during the most recent issue): Mon Apr 20 13:46:05 2015 notice OTRS-CGI-61 User: dritchie authentication ok (Method: sha256, REMOTE_ADDR: 10.0.255.00). Mon Apr 20 13:40:04 2015 info OTRS-otrs.PostMasterMailbox.pl-61 POP3S: Fetched 2 email(s) from xxxxxxxxxxxxxxxxxxxxxxxx Mon Apr 20 13:40:04 2015 info OTRS-otrs.PostMasterMailbox.pl-61 Sent agent 'NewTicket' notification to 'dritchie@mtcperformance.com'. Mon Apr 20 13:40:04 2015 info OTRS-otrs.PostMasterMailbox.pl-61 New Ticket [61008259/WARNING. Someon] created (TicketID=8257,Queue=Raw,Priority=3 normal,State=new) /var/log/http/access_log (covering the same time frame): **my successful logout from earlier** 10.0.255.00 - - [20/Apr/2015:13:09:52 -0500] "GET /otrs-web/js/js-cache/ModuleJS_ecdc48f1f171c4862d8934c0aba66b85.js HTTP/1.1" 200 368 "http://ticketxx.yyyy.com/otrs/index.pl?Action=Logout;ChallengeToken=6uM1wcJW...;" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.18 Safari/537.36" ** My attempt to login** 10.0.255.00 - - [20/Apr/2015:13:41:45 -0500] "GET / HTTP/1.1" 302 26 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.18 Safari/537.36" **NOTE: this is my testing if the apache server is responding correctly** 10.0.255.00 - - [20/Apr/2015:13:42:10 -0500] "GET /server-status?refresh=25 HTTP/1.1" 200 6482 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.18 Safari/537.36" **successful login** 10.0.255.00 - - [20/Apr/2015:13:46:02 -0500] "GET /otrs/index.pl?Action=AdminPostMasterFilter;Subaction=AddAction HTTP/1.1" 200 3594 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.18 Safari/537.36" /var/log/http/error_log nothing Notice that the Apache log shows an attempt to log in at 13:41:45 while the System Log (from the admin page of the website) shows nothing until the successful login at 13:46:05 (also shown at 13:46:02 in the Apache Access Log). Any Ideas? Thank You Dennis From: otrs-bounces@otrs.org [mailto:otrs-bounces@otrs.org] On Behalf Of David Boyes Sent: Monday, April 20, 2015 9:58 AM To: User questions and discussions about OTRS. Subject: Re: [otrs] question The 15 minutes can be to get a logon screen. Or if already logged in to switch screens. That sounds more like a resource constraint problem on the server running the OTRS code than a database problem - OTRS is a relatively large application, and can be fairly demanding on application server resources. Try monitoring CPU, swap and memory utilization at 5 minute intervals on the OTRS server for a few days and see if there's any correlation to the problem periods - if the OTRS server is thrashing, the database response is irrelevant. Eliminate that possibility first, and then start looking at database performance.
Is it possible that your Apache is configured to try to do reverse lookups on incoming connections, and you're getting timeouts from DNS on the reverse lookup? Or that one or more of your DNS servers is not responding? At least in that example, if something is trying to do reverse lookups, the 5 minute delay would be about right if one or more of the DNS servers weren't reachable/not responding.
On 15-04-20 03:15 PM, Ritchie, Dennis wrote:
<quote> That sounds more like a resource constraint problem on the server </quote>
I agree but see below (I’ve munged some of the information):
Are these top stats from the OTRS server or the MySQL server? I think it would help if you could share some stats on the MySQL server. I typically set the sysstat cronjob to 1 minutes and use kSar to graph the results. But top output may be ok. Well top with the output of "vmstat 1 15". Thanks,
They were the stats from the OTRS Server not the SQL server. However, I think that I may have found my issue. I am using a POP3 Account to take in email and the default for Re-authorizing to the server was 20 emails. Ticket -> Core::PostMaster::PostMasterReconnectMessage It seems that my POP3 provider is not particularly responsive and the repeated re-authorization requests was causing the system slowdown (I'm still testing to be sure). I increased the value from 20 to 999 (Max) and OTRS has been much more stable. I have only had to restart it 1 time since the change and that was almost 1 full month later. I have to wonder why this would keep the system from responding. Shouldn't the postmaster script be separate from the rest of the system and running on it's own thread(s)? Apache as I mentioned was running and accessible the entire time. So thank you to all who have tried to help me with this issue. Thank You Dennis -----Original Message----- From: otrs-bounces@otrs.org [mailto:otrs-bounces@otrs.org] On Behalf Of Ugo Bellavance Sent: Wednesday, May 20, 2015 7:13 PM To: User questions and discussions about OTRS. Subject: Re: [otrs] question On 15-04-20 03:15 PM, Ritchie, Dennis wrote:
<quote> That sounds more like a resource constraint problem on the server </quote>
I agree but see below (I've munged some of the information):
Are these top stats from the OTRS server or the MySQL server? I think it would help if you could share some stats on the MySQL server. I typically set the sysstat cronjob to 1 minutes and use kSar to graph the results. But top output may be ok. Well top with the output of "vmstat 1 15". Thanks, --------------------------------------------------------------------- OTRS mailing list: otrs - Webpage: http://otrs.org/ Archive: http://lists.otrs.org/pipermail/otrs To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs
On 15-05-21 11:26 AM, Ritchie, Dennis wrote:
They were the stats from the OTRS Server not the SQL server.
If you'd like some help, you should provide the information about the MySQL server...
However, I think that I may have found my issue. I am using a POP3 Account to take in email and the default for Re-authorizing to the server was 20 emails. Ticket -> Core::PostMaster::PostMasterReconnectMessage
It seems that my POP3 provider is not particularly responsive and the repeated re-authorization requests was causing the system slowdown (I'm still testing to be sure). I increased the value from 20 to 999 (Max) and OTRS has been much more stable. I have only had to restart it 1 time since the change and that was almost 1 full month later.
I never have to restart anything with OTRS.
I have to wonder why this would keep the system from responding. Shouldn't the postmaster script be separate from the rest of the system and running on it's own thread(s)? Apache as I mentioned was running and accessible the entire time.
The postmaster script should not cause any issues, unless it is locking a database table.
So thank you to all who have tried to help me with this issue.
Thank You
Dennis
-----Original Message----- From: otrs-bounces@otrs.org [mailto:otrs-bounces@otrs.org] On Behalf Of Ugo Bellavance Sent: Wednesday, May 20, 2015 7:13 PM To: User questions and discussions about OTRS. Subject: Re: [otrs] question
On 15-04-20 03:15 PM, Ritchie, Dennis wrote:
<quote> That sounds more like a resource constraint problem on the server </quote>
I agree but see below (I've munged some of the information):
Are these top stats from the OTRS server or the MySQL server? I think it would help if you could share some stats on the MySQL server. I typically set the sysstat cronjob to 1 minutes and use kSar to graph the results. But top output may be ok. Well top with the output of "vmstat 1 15".
Thanks,
--------------------------------------------------------------------- OTRS mailing list: otrs - Webpage: http://otrs.org/ Archive: http://lists.otrs.org/pipermail/otrs To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs --------------------------------------------------------------------- OTRS mailing list: otrs - Webpage: http://otrs.org/ Archive: http://lists.otrs.org/pipermail/otrs To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs
participants (4)
-
David Boyes -
Friedrich Kölbel -
Ritchie, Dennis -
Ugo Bellavance