System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7
-
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Maybe reboot post maintenance so that we know that we have a fresh, clean system and see what it does.
Done... awaiting system to come back online.
-
Session-db.wal went from 78,403,632 to 4,128,272 !!!
App_Data]$ ls -l total 28836 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25284608 Dec 17 02:17 Session.db -rw-r--r-- 1 root root 32768 Dec 17 02:17 Session.db-shm -rw-r--r-- 1 root root 4128272 Dec 17 02:17 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e 6cea952.xml -rw-r--r-- 1 root root 21908 Dec 16 17:24 User.xml -
That's quite a change.
-
No heavy CPU or memory on restart either.
-
Of note - the only 'difference' was the selection of Maintenance Plan Actions:
Previously only the Delete Ended Support Sessions and Compact Database Files were selected. The boxed items were not.
-
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-
SAR
r this morning - near as I can tell, no one is on SC currently.$ uptime 10:36:37 up 22:38, 1 user, load average: 0.05, 0.17, 0.22 [ ]$ sar Linux 3.10.0-327.10.1.el7.x86_64 ( ) 12/18/2016 _x86_64_ (1 CPU) 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 1.89 0.00 0.36 0.09 0.06 97.60 12:20:01 AM all 1.03 0.00 0.33 0.03 0.03 98.57 12:30:01 AM all 1.83 0.00 0.36 0.07 0.05 97.68 12:40:01 AM all 0.89 0.00 0.29 0.03 0.05 98.73 12:50:01 AM all 1.68 0.00 0.35 0.08 0.04 97.85 01:00:01 AM all 1.03 0.00 0.34 0.07 0.03 98.54 01:10:01 AM all 2.30 0.00 0.45 0.13 0.06 97.05 01:20:01 AM all 0.79 0.00 0.29 0.05 0.03 98.84 01:30:01 AM all 1.82 0.00 0.36 0.09 0.08 97.65 01:40:01 AM all 0.71 0.00 0.28 0.02 0.03 98.97 01:50:01 AM all 1.78 0.00 0.35 0.11 0.07 97.70 02:00:01 AM all 0.74 0.00 0.29 0.06 0.03 98.88 02:10:02 AM all 1.76 0.00 0.36 0.09 0.07 97.72 02:20:01 AM all 1.16 0.00 0.33 0.04 0.04 98.43 02:30:01 AM all 1.90 0.00 0.35 0.08 0.11 97.56 02:40:01 AM all 0.76 0.00 0.28 0.02 0.05 98.88 02:50:01 AM all 1.80 0.00 0.36 0.24 0.08 97.52 03:00:01 AM all 0.69 0.00 0.28 0.03 0.05 98.95 03:10:01 AM all 1.86 0.02 0.36 0.08 0.07 97.61 03:20:01 AM all 0.82 0.00 0.32 0.02 0.04 98.79 03:30:01 AM all 1.84 0.00 0.34 0.09 0.05 97.67 03:40:01 AM all 0.74 0.00 0.28 0.03 0.04 98.91 03:50:01 AM all 1.81 0.00 0.35 0.09 0.06 97.70 04:00:01 AM all 0.82 0.00 0.29 0.03 0.04 98.82 04:10:01 AM all 2.42 0.00 0.44 0.09 0.10 96.96 04:20:01 AM all 0.77 0.00 0.29 0.03 0.05 98.86 04:30:01 AM all 1.84 0.00 0.37 0.09 0.10 97.61 04:40:01 AM all 0.75 0.00 0.30 0.06 0.06 98.83 04:50:01 AM all 1.84 0.00 0.36 0.09 0.08 97.63 05:00:01 AM all 0.81 0.00 0.29 0.02 0.05 98.83 05:10:01 AM all 2.17 0.00 0.43 0.09 0.11 97.21 05:20:01 AM all 0.85 0.00 0.29 0.16 0.04 98.67 05:30:01 AM all 1.79 0.00 0.38 0.09 0.07 97.67 05:40:01 AM all 0.86 0.00 0.32 0.03 0.08 98.72 05:50:01 AM all 1.94 0.00 0.46 0.16 0.12 97.32 06:00:01 AM all 0.80 0.00 0.33 0.02 0.05 98.80 06:10:01 AM all 1.79 0.00 0.37 0.33 0.08 97.42 06:20:01 AM all 0.93 0.00 0.34 0.02 0.09 98.62 06:30:01 AM all 1.82 0.00 0.36 0.08 0.09 97.65 06:40:02 AM all 0.76 0.00 0.29 0.03 0.06 98.87 06:50:01 AM all 1.87 0.00 0.37 0.09 0.11 97.56 07:00:01 AM all 0.82 0.00 0.30 0.03 0.05 98.79 07:10:02 AM all 2.23 0.00 0.46 0.08 0.15 97.09 07:20:01 AM all 0.82 0.00 0.32 0.03 0.09 98.74 07:30:01 AM all 2.84 0.00 0.75 0.09 0.12 96.20 07:40:01 AM all 3.05 0.00 0.30 0.03 0.09 96.54 07:50:01 AM all 24.73 0.32 0.62 0.12 0.15 74.07 08:00:01 AM all 6.52 0.00 0.34 0.04 0.07 93.03 08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 25.18 0.00 0.58 0.07 0.16 74.01 08:20:01 AM all 7.17 0.00 0.34 0.03 0.10 92.37 08:30:01 AM all 25.70 0.00 0.52 0.12 0.23 73.42 08:40:01 AM all 6.75 0.00 0.34 0.03 0.13 92.75 08:50:01 AM all 24.80 0.00 0.50 0.07 0.17 74.45 09:00:01 AM all 7.47 0.00 0.34 0.02 0.08 92.08 09:10:01 AM all 28.54 0.00 0.63 0.09 0.48 70.27 09:20:01 AM all 7.15 0.00 0.36 0.17 0.17 92.16 09:30:01 AM all 26.84 0.00 0.54 0.09 0.34 72.20 09:40:01 AM all 8.76 0.00 0.36 0.07 0.27 90.54 09:50:01 AM all 26.93 0.00 0.53 0.07 0.31 72.15 10:00:01 AM all 8.60 0.00 0.35 0.03 0.18 90.84 10:10:01 AM all 28.29 0.00 0.63 0.08 0.47 70.53 10:20:01 AM all 9.19 0.00 0.33 0.08 0.28 90.12 10:30:01 AM all 27.27 0.00 0.52 0.08 0.32 71.81 Average: all 5.79 0.01 0.38 0.07 0.11 93.64 -
-
~]$ cd /opt/screenconnect//App_Data/ [gene@dny-lnx-sc App_Data]$ ls -l total 49048 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 24989696 Dec 18 10:26 Session.db -rw-r--r-- 1 root root 65536 Dec 18 10:39 Session.db-shm -rw-r--r-- 1 root root 25082592 Dec 18 10:39 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 17 10:41 User.xml -
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-rw-r--r-- 1 root root 25,082,592 Dec 18 10:39 Session.db-wal -
So it is climbing back up and fast. Show that to support.
-
Memory is up to 398M used now.
SAR is bouncing around as well -
08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 25.18 0.00 0.58 0.07 0.16 74.01 08:20:01 AM all 7.17 0.00 0.34 0.03 0.10 92.37 08:30:01 AM all 25.70 0.00 0.52 0.12 0.23 73.42 08:40:01 AM all 6.75 0.00 0.34 0.03 0.13 92.75 08:50:01 AM all 24.80 0.00 0.50 0.07 0.17 74.45 09:00:01 AM all 7.47 0.00 0.34 0.02 0.08 92.08 09:10:01 AM all 28.54 0.00 0.63 0.09 0.48 70.27 09:20:01 AM all 7.15 0.00 0.36 0.17 0.17 92.16 09:30:01 AM all 26.84 0.00 0.54 0.09 0.34 72.20 09:40:01 AM all 8.76 0.00 0.36 0.07 0.27 90.54 09:50:01 AM all 26.93 0.00 0.53 0.07 0.31 72.15 10:00:01 AM all 8.60 0.00 0.35 0.03 0.18 90.84 10:10:01 AM all 28.29 0.00 0.63 0.08 0.47 70.53 10:20:01 AM all 9.19 0.00 0.33 0.08 0.28 90.12 10:30:01 AM all 27.27 0.00 0.52 0.08 0.32 71.81 10:40:01 AM all 8.15 0.00 0.33 0.07 0.23 91.22 10:50:01 AM all 25.58 0.00 0.46 0.07 0.27 73.61 11:00:01 AM all 41.85 0.00 0.66 0.14 0.29 57.06 11:10:01 AM all 26.54 0.00 0.62 0.10 0.47 72.27 11:20:01 AM all 8.44 0.00 0.39 0.03 0.21 90.92 11:30:01 AM all 43.08 0.00 0.68 0.15 0.58 55.50 11:40:01 AM all 27.83 0.00 0.52 0.11 0.47 71.07 11:50:01 AM all 24.17 0.00 0.53 0.07 0.38 74.84 12:00:01 PM all 8.93 0.00 0.42 0.04 0.31 90.30 12:10:01 PM all 26.29 0.00 0.60 0.10 0.40 72.61 12:20:01 PM all 9.36 0.00 0.61 0.05 0.17 89.80 12:30:01 PM all 23.48 0.00 0.66 0.07 0.19 75.60 12:40:01 PM all 9.61 0.00 0.50 0.06 0.21 89.63 12:50:01 PM all 23.44 0.00 0.62 0.06 0.18 75.70 01:00:01 PM all 8.63 0.00 0.48 0.03 0.10 90.76 01:10:01 PM all 24.10 0.00 0.70 0.07 0.21 74.93 Average: all 8.90 0.00 0.41 0.07 0.15 90.46 -
Yup that CPU is WAY too high.
-
Curious - system did a restart this evening.
05:05:15 PM LINUX RESTART 05:10:01 PM CPU %user %nice %system %iowait %steal %idle 05:20:01 PM all 3.56 0.00 0.15 0.02 0.09 96.19 05:30:01 PM all 14.36 0.00 0.23 0.09 0.07 85.25 05:40:01 PM all 4.74 0.00 0.16 0.03 0.09 94.99 05:50:01 PM all 16.36 0.00 0.26 0.14 0.11 83.13 06:00:01 PM all 3.68 0.00 0.13 0.02 0.06 96.12 06:10:01 PM all 16.09 0.00 0.27 0.10 0.13 83.41 06:20:01 PM all 4.60 0.00 0.24 0.13 0.09 94.96 06:30:01 PM all 15.20 0.00 0.24 0.32 0.11 84.14 06:40:01 PM all 3.54 0.00 0.13 0.01 0.05 96.27 06:50:01 PM all 15.19 0.00 0.24 0.09 0.07 84.40 07:00:01 PM all 5.10 0.00 0.14 0.02 0.07 94.68 07:10:01 PM all 15.11 0.00 0.29 0.13 0.11 84.36 07:20:01 PM all 3.91 0.00 0.12 0.03 0.06 95.89 07:30:01 PM all 14.58 0.00 0.24 0.08 0.08 85.02 07:40:01 PM all 4.20 0.00 0.18 0.02 0.05 95.55 07:50:01 PM all 14.34 0.00 0.26 0.07 0.08 85.25 Average: all 9.65 0.00 0.20 0.08 0.08 89.98DB is still on the upside for 25MB
]$ ls -l total 49596 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25583616 Dec 18 19:48 Session.db -rw-r--r-- 1 root root 32768 Dec 18 19:50 Session.db-shm -rw-r--r-- 1 root root 25082592 Dec 18 19:50 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 17 10:41 User.xml -
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
-
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
what data will you lose?
-
@Dashrender said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
what data will you lose?
We should not lose any data. Just some down time while the process is completeed.
-
Process done:
sudo service screenconnect stopmv /opt/screenconnect/Session.db /opt/screenconnect/Session.db.baksudo service screenconnect start -
]$ ls -l total 29724 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 1232896 Dec 19 20:14 Session.db -rw-r--r-- 1 root root 24956928 Dec 19 20:11 Session.db.bak -rw-r--r-- 1 root root 32768 Dec 19 20:18 Session.db-shm -rw-r--r-- 1 root root 4128272 Dec 19 20:18 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 19 20:13 User.xml -
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-rw-r--r-- 1 root root 25,082,592 Dec 18 10:39 Session.db-walSession.db-wal is not the right file to be looking at... Should be looking at session.db
