System is responding slowly

When devices are not connected directly to the internet, it is possible it goes thru routers, switches, proxy servers and other possible network routes to access the Wavelet EMP on the cloud. There are a few ways to check and detect if the problems are caused by the Local Area Network:

• Non-technical approach
  • Getting help from another users from another location to access the website
  • Use a different device that directly connect to the internet, for example you are having problems with your computer accessing the website, but you can test with your phone (without connecting to Wifi, use direct internet connection, 3G etc)
• Using Ping
  • Ping your router IP address (not the google and not other external websites / servers)
• Monitoring the network performance on the networking devices
  • this hyperlink provides tutorial for D-Link routers, but other routers should be similar as well, refer to the respective user manual for other routers.
• Other devices within the same local area network
  • Sometimes, there's no problem with your device or routers, but there could be another device within the local area network that is infected by Virus, and consuming significant amount of bandwidth. It is also possible that somewhere in the LAN (Local Area Network), there's another users installed some applications like Torrent that download and share large files, or watching youtube, or consuming some live streaming content.

• A server network, or simply a network, is a collection of computers and other hardware components interconnected by communication channels that allow sharing of resources and information.
  
  
•  Basic knowledge of Networking, You don't need to be a CISCO Certified Engineer, but some basic knowledge about IP address, Sub-net mask, Gateway, DNS Server, Port Forwarding, Internal IP Address, External IP address of the modem or Router, how DYNDNS
  • http://www.cctvcamerapros.com/LinkSys-Port-Forwarding-s/122.htm
  • http://pigtail.net/LRP/portforward.html
  • http://documentation.netgear.com/dg834n/enu/202-10197-02/Advanced.7.3.html
    
    
• Majority of customers located at AWS, to check the server network:
  • Click the instance -> monitoring. Refer to network in and network out graph and observe
    
  •  

    If the network out too high, use netstat to observe which IP that causing high network out. kill and chmod the process will help

  • Power up new instance and migrate the customer database using the new and safer instance security. This to be done after working hours
    
    
• Troubleshot the server network
  
  • Check IP Config
    • ifconfig (for linux customer)
    • ipconfig (for windows customer)
  • Set Static IP via terminal
  • Refer to :
    • https://www.howtoforge.com/linux-basics-set-a-static-ip-on-ubuntu
      
  • Solve Session Time Out error in Firefox
  • Refer to :
    • http://support.filesanywhere.com/hc/en-us/articles/214950806-Resolving-Session-Time-Out-Errors-
      
  • Linux System Security
  • Refer to :
    • https://linas.org/linux/secure.html
      

To trouble Shoot Server Slow Unresponsive, always refer to basic guide at /wiki/spaces/WU/pages/11568251

The steps to checks:

•  Server Log
  • As root, vim /usr/java/jboss/server/default/log/server.log
  • Scroll to the last line (shift+G) of the server log
  • Check the exception message, copy the exception message, find the 1st occurrence of the exception (but disregard those regular exceptions for example the one caused by PDF printing “.. unable to forward…”
  • If the message looks something like ‘.. I/O exception…’ or ‘.. unable to connect to the database …’, check the database right away (refer to point 9 Server Database Server)
  • If the message looks something like ‘.. out of memory…’, check the Java Memory Allocation
    
    
• CPU, Memory usage

  • To check the usage, type :

    top

  • You can display the details for the running processes, by typing c
  • You can sort the processes by highest memory usage first, by typing m
  • If the memory usage is mainly by PosgreSQL, refer to point 9

  • If the CPU usage is high (>= 90%), type this command to check 3 to 5 process that consume high CPU:

    ps aux | sort -nrk 3,3 | head -n 5

  • Kill the process that have high CPU

    kill -9 <process id>

  • Usually, virus inside /tmp folder, if find virus inside /tmp folder

  • Directly remove the virus by :

    cd /tmp rm -rf /tmp/*

  • Put this command on crontab -e at other user instead of root user, because hackers usually attack by using root.

  • For example user emp/empbackup/opswork. Please make sure the user that you choose to put crontab in is sudoers without having to put sudo password.

    

  • Put this command on crontab -e at other user instead of root user, because hackers usually attack by using root (emp, empbackup, opswork)

  • This command will delete file inside /tmp folder every one minute

    */1 * * * * sudo rm -rf /tmp/*

  • Need to make sure doesn't need any password when type command:
    

    sudo su -

• Operating System log
  

  • it could be caused by hardware failure, which will be recorded into the O/S log. You can access the O/S log at /var/log/message
    
    

• JBoss status
  • This is no brainer, just make sure JBoss is actually running,
  • And ONLY 1 instance of JBoss is running (if you are using docker container /wiki/spaces/WU/pages/13042094, /wiki/spaces/WU/pages/38928514)

  • run command :

    ps aux | grep java

    
    

• Jboss memory allocation

  • If RAM usage is very high, check the memory allocation for jboss : /usr/java/jboss-4.0.1RC2/server/default/deploy/jbossweb-tomcat50.sar/server.xml

  • See if the xms is too low or xmx too high - compared to the RAM size.

• In most cases, customer is inpatient with server down, immediately offer to restart their server when idle in transaction happened. After restarting server, check the server log and troubleshot what was happened.

• Checking idle transaction :

  select * from pg_stat_activity where (state = 'idle in transaction')

  select count(*), state, datname from pg_stat_activity where state = 'idle in transaction' group by datname, state order by count(*) desc

  
  

  • If jboss-stop doesn't run anything

  • use :

    ps aux|grep java kill -9 2304

    
    
    

    etc/init.d/postgresql-9.2.4 restart jboss-start

  • Check previous date server log : 

    cd /usr/java/jboss/server/default/log/ ls -lhrt

    • Check the specific date of the server log
• Missing run.sh and shutdown.sh from server
  • Refer to /wiki/spaces/WU/pages/37945457
• CPU hits 100%
  • We have seen several times that customer CPU has reached almost 100%

    • Get the PID: the very first step is to know what is the Java process ID

      ps aux|grep java

    • Get CPU usage per each LWP related to the main process,

      ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args|grep {pid} > CUSTOMERNAME_DATE_lwpthread.txt

    • Generate a java thread dump

      jstack -l {pid}  > /root/CUSTOMERNAME_DATE__threadDump.txt

    • Open CUSTOMERNAME_DATE_lwpthread.txt and take note lwp on the second column
    • Convert your LWP id from DEC to HEX http://www.binaryhexconverter.com/decimal-to-hex-converter
    • 
    • Open CUSTOMERNAME_DATE_threadDump.txt, and locate HEX value you will find something similar
    • 
    • Reference: 
      • http://javadrama.blogspot.my/2012/02/why-is-java-eating-my-cpu.html
      • https://blog.fastthread.io/2016/06/06/how-to-take-thread-dumps-7-options/

• Check to make sure you are able to connect to the database
  • If you can’t, if it is a
    • App / DB setup
    • Check if network is up (refer to above points 1 to 6)
    • Check if database server is up
      • If you are not able to connect to the wsemp database, most probably your postgres has not started yet.
      • run command as root, /etc/init.d/postgresql-9.2.4 start and try psql command
        
    • Check if PostgreSQL is running
    • Check if the database server has rebooted, by using last. If this is occurring not during the cron script time (6am), ALERT FAULTY SERVER FAN!!!!
    • Combo setup
      
      
• To look at running query

  • To look at running query

    SELECT pg_stat_get_backend_pid(s.backendid) AS procpid, pg_stat_get_backend_activity(s.backendid) AS current_query FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s;

    Analyzing Idle In Transaction, 

• Postgresql Log

  • Located at :

    /var/lib/pgsql/data/serverlog

  • Errors are usually very apparent, they will tell you it's an error

  • You can locate the related table / query that might be causing the problem

    
    

• High Availability Load Balancing and Replication for PostgreSQL
• Performance Tuning
  
  
• When customer complain their reports run very slow, but after you repopulate the database to your localhost no such thing as slowness occurred. What you need to do
  • Check the whether the VACUMM is running. 

    • To check as empbackup users. run :

      vim backup.log

      you need to make sure the latest backup log shows the VACUMM is done.

      
      
      

  • Inform customer that you will repopulate database again at night. In same cases, the slowness is gone after we re-populate their database. This can't be done at working hours unless customer ask you to
•  RDS connection
• Can check on staging-account (RDS)
• https://staging-wavelet.signin.aws.amazon.com/console
  
• Click on "Show monitoring"
• Select "DB Connection"
  
• 
  
  
  
  
• Need to make sure Cpu, Memory and Storage are not more than red line
• If not more than red line, need to check Logs
• If found out any process that weird, for example :

  May 24, 2017 at 7:59:49 PM UTC+8 2017-05-24 11:00:15 UTC:172.31.26.7(38486):sbs@sbs:[59139]:ERROR: update or delete on table "bl_fi_mst_entity_hdr" violates foreign key constraint "app_master_entity_line_app_master_entity_header" on table "bl_fi_mst_entity_line" 2017-05-24 11:00:15 UTC:172.31.26.7(38486):sbs@sbs:[59139]:DETAIL: Key (guid)=(6DC2A9FE-364A-4C71-87F3-2C773CDF49E2) is still referenced from table "bl_fi_mst_entity_line". 2017-05-24 11:00:15 UTC:172.31.26.7(38486):sbs@sbs:[59139]:STATEMENT: DELETE FROM bl_fi_mst_entity_hdr WHERE name IN (SELECT ety_hdr.name AS ety_hdr_name_indicator FROM bl_fi_mst_entity_hdr AS ety_hdr GROUP BY ety_hdr.name HAVING COUNT(1) > 1) AND guid NOT IN (SELECT MIN(ety_hdr.guid) AS min_unique_guid FROM bl_fi_mst_entity_hdr AS ety_hdr GROUP BY ety_hdr.name HAVING COUNT(1) > 1) May 24, 2017 at 8:59:34 AM UTC+8 2017-05-24 12:00:16 UTC:172.31.26.7(38486):sbs@sbs:[59139]:ERROR: update or delete on table "bl_fi_mst_entity_hdr" violates foreign key constraint "app_master_entity_line_app_master_entity_header" on table "bl_fi_mst_entity_line" 2017-05-24 12:00:16 UTC:172.31.26.7(38486):sbs@sbs:[59139]:DETAIL: Key (guid)=(6DC2A9FE-364A-4C71-87F3-2C773CDF49E2) is still referenced from table "bl_fi_mst_entity_line". 2017-05-24 12:00:16 UTC:172.31.26.7(38486):sbs@sbs:[59139]:STATEMENT: DELETE FROM bl_fi_mst_entity_hdr WHERE name IN (SELECT ety_hdr.name AS ety_hdr_name_indicator FROM bl_fi_mst_entity_hdr AS ety_hdr GROUP BY ety_hdr.name HAVING COUNT(1) > 1) AND guid NOT IN (SELECT MIN(ety_hdr.guid) AS min_unique_guid FROM bl_fi_mst_entity_hdr AS ety_hdr GROUP BY ety_hdr.name HAVING COUNT(1) > 1)

• Those above error happened where the query is trying to delete data with guid 6DC2A9FE-364A-4C71-87F3-2C773CDF49E2 for one hour and it violates constraint.
  
• It might come from ETL and DMS server.
• How to handle :
• Refer to :
  • /wiki/spaces/WM/pages/130565063
    
• RDS
  • Check Queue Depth chart, if the maximum reaches 40. Some server might have problem with "Unable to connect to database"
  • 
  • Go to RDS postgresql log file and find PSQL exception and long running queries. You need get help from development team to analyse the queries and how to prevent it
  • 
  • 
  • 

  • Observe RDS db connection and running queries /wiki/spaces/WM/pages/130117823

Server slowness can be found only from specific module from EMP caused by Inefficient code or SQL queries

• Get from customer which module causing slowness and test from your personal PC or Laptop whether the module is really slow. Run command showlog, observe whether there is any exception error found. If you are not confident on the code, get any programmer to sit beside you and help to troubleshoot it.
  
  • Variety of exceptions:
    • org.postgresql.util.PSQLException. Solutions: copy the query before that error appeared and report to programming team to fix the SQL query.
      
      
    • nullpointerExecption. Solutions: Repopulate the database to your localhost. Run the debug or your eclipse and found which table or queries causing null pointer exception. In most cases, error will be solved if you find which table has null data and update it to some values depends on the "Type". Example, if the Type is integer, you can update to 0, If it is string, update to '' or empty string
      
• Common cases is that Modules is missing from EMP. Please refer to below wiki to fix the problem. 
  • GUID exception error, UnknownHostException: Menu toolbar missing after reboot