| No | Possible Root Cause | Troubleshooting Guide | Solutions |
|---|
| 1 | Client Device | The fastest way to check if the slowness is caused by the client device, is to access the same website / functions with multiple other devices. If Accessibility from other devices are normal, then the probability of slowness caused by the slowness in the device itself is almost certain. Regardless of whether it is mobile or desktop devices, see below for more steps to identify the root cause: - Android Devices
- iOS
- There are many system monitoring Apps in the iOS world as well, examples as below:
- Mac
- Out of the box, we could use the Activity Monitor that comes with Mac OS
- Windows
- There are many different versions of Windows out there, and there are plenty of resources for different versions of Windows, some of the useful resources as below:
Apart from System Performance monitoring, it is never alien to our technical support team that some of the client devices that have been infected by Virus is known to perform badly. Usually, when we detected high CPU usage, and high Network Bandwidth, we would also be recommending the users to install anti-virus software to detect the potential security threat to the devices. Another possibility is the web browser. When it comes to web-browser usage, settings, the following areas could be checked: - DNS settings
- Too many tabs
- Cache
- Proxy Settings
| Depends on the root cause from the troubleshooting of the client device, users could upgrade the RAM, Processor, or perform anti-virus scanning accordingly. At the time of reading, the following guide may be out-dated, but google is your best friends, there are plenty of resources available online to resolve the problems that you are having with optimizing your devices. - Android Devices
- iOS
- Mac
- Windows
|
| 2 | Client Local Area Network | When devices are not connected directly to the internet, it is possible it goes thru routers, switches, proxy servers and other possible network routes to access the Wavelet EMP on the cloud. There are a few ways to check and detect if the problems are caused by the Local Area Network: - Non-technical approach
- Getting help from another users from another location to access the website
- Use a different device that directly connect to the internet, for example you are having problems with your computer accessing the website, but you can test with your phone (without connecting to Wifi, use direct internet connection, 3G etc)
- Using Ping
- Ping your router IP address (not the google and not other external websites / servers)
- Monitoring the network performance on the networking devices
-
- this hyperlink provides tutorial for D-Link routers, but other routers should be similar as well, refer to the respective user manual for other routers.
- Other devices within the same local area network
- Sometimes, there's no problem with your device or routers, but there could be another device within the local area network that is infected by Virus, and consuming significant amount of bandwidth. It is also possible that somewhere in the LAN (Local Area Network), there's another users installed some applications like Torrent that download and share large files, or watching youtube, or consuming some live streaming content.
| |
| 3 | Client Internet Connectivity | On the client side, for easier troubleshoot. Please ask customer to use right tools such as Teamviewer. Top 3 reasons from client side - The Obvious: The URL has a typo.
- Ask the user to double-check the URL. This may seem obvious but it is very critical that the user puts in the correct URL. It is better to ask the user to send you the URL so that you can examine and try it yourself. There are three things to note in the URL.
- Is the protocol correct? (i.e) http. Your Web Server admin could have very well blocked port 80 altogether
- Is the domain name and the resource being accessed correctly spelled?
- Is the Port number correct ? (if any). The browser will default http to 80 or http to 8080.
- The user’s PC does not have Network connection
- Ask the user to access other sites to make sure his PC is in the Network. If he cannot access any sites, chances are his PC is having some issues.
- The user’s PC is not able to resolve names (DNS)
- It is possible that the user’s PC is unable to resolve any names due to DNS issues. Have the user execute “nslookup <domain name>” in command prompt and ensure that he is able to resolve the domain name to an IP.
- The client router's problem
- Advise client to find internal IT expert to fix their router until they able to connect to internet. Kindly inform customer we are not supporting internal problem such as router is reset, LAN cable doesn't work, problem with WIFI. We only support everything that related to EMP server and Port forwarding.
- If the server is accessible internally, but not accessible externally. You need to ask customer to open router page and login to the router's page and check the Port Forwarding setup. Get the router model and brand and proceed to research the configuration from google. If you can get their internal IT to fix port forwarding, that will be better.
- The client's router is congested
- Root cause : A large number of network broadcasts, A large number of broadcasts can be an indication of a misconfigured or faulty NIC, host, switch, software, driver, etc. and can also be an indication of a malware infection somewhere in the network
- High usage from user's client internally such as downloading and buffering behavior that consumes too much connectivity content.
- We can advise customer to "restart" their modem and router. Remove the power cord from the modem and/or router, wait at least 10 seconds, and then plug the modem and/or router back in.
| Troubleshoot Internet connection problems
http://windows.microsoft.com/en-us/windows-vista/troubleshoot-internet-connection-problems
Guide to set DNS
https://developers.google.com/speed/public-dns/docs/using?hl=en |
| 4 | Public Internet | If you are connected from "Public Internet", you might experienced having difficulties of accessing URL which causes by: - Enhanced Protected Mode
- Browser History
- Browser Add-On
- Proxy and DNS Settings
- Browser need to be reset
- Check whether a third-party service, program, or anti-virus is conflicting with browser
- Temporarily disabling the firewall
- Updating older drivers and editing registry key TabProcGrowth
- Checking Windows Updates for drivers
- Restore or refresh your PC
| Refer to this wiki for solutions https://support.microsoft.com/en-us/kb/956196 |
| 5 | Data Center Network | Always refer to complete server list https://docs.google.com/spreadsheets/d/1nfHlOJYQhhxe_aMSOLJd6zC1cL9dptmPPf8t7zMwllE/edit#gid=0 | Need to ask from data center - Internet connectivity of data center
- Customer server condition.
- ssh port is accessible?
- backup database can be restored and manually restored to wavelet cloud or AWS
AWS confluences: http://intranet.wavelet.asia/projects/cloud-amazon/wiki/Launching_new_EMP_instance_on_Amazon /wiki/spaces/WU/pages/36798614 /wiki/spaces/WU/pages/22839391 /wiki/spaces/WU/pages/28803771 Azure confluences: /wiki/spaces/WU/pages/11567303 |
| 6 | Server Network | For AWS server network, you may click the instance -> monitoring. Refer to network in and network out graph and observer .png?version=1&modificationDate=1450775869750&cacheVersion=1&api=v2&width=538&height=250)
| If the network out too high, use netstat to observe which IP that causing high network out. kill and chmod the process will help |
| 7 | Server Operating Systems | Server Operating Systems : Ubuntu, Centos, Amazon Linux AMI | Useful LINUX command http://www.tecmint.com/useful-linux-commands-for-system-administrators/ The different between Ubuntu and Centos Ubuntu : apt-get Centos : yum |
| 8 | Server Application Server | To trouble Shoot Server Slow Unresponsive, always refer to basic guide at /wiki/spaces/WU/pages/11568251 The steps to checks: - Server Log
- As root, vim /usr/java/jboss/server/default/log/server.log
- Scroll to the last line (shift+G) of the server log
- Check the exception message, copy the exception message, find the 1st occurrence of the exception (but disregard those regular exceptions for example the one caused by PDF printing “.. unable to forward…”
- If the message looks something like ‘.. I/O exception…’ or ‘.. unable to connect to the database …’, check the database right away (refer to point 9 Server Database Server)
- If the message looks something like ‘.. out of memory…’, check the Java Memory Allocation, refer to
- CPU, Memory usage
- Operating System log
- JBoss status
Jboss memory allocation If RAM usage is very high, check the memory allocation for jboss : /usr/java/jboss-4.0.1RC2/server/default/deploy/jbossweb-tomcat50.sar/server.xml See if the xms is too low or xmx too high - compared to the RAM size.
- In most cases, customer is inpatient with server down, immediately offer to restart their server when idle in transaction happened. After restarting server, check the server log and troubleshot what was happened.
http://intranet.wavelet.asia/projects/tech/wiki/Trouble_Shoot_Server_Slow__Unresponsive http://intranet.wavelet.asia/projects/tech/wiki/Server_Slow_Issue Missing run.sh and shutdown.sh Auto add run.sh and shutdown.sh using rc.local script | Check running process using these command and observe showlog : Output the running server log top : List processes running on the system free -m : check RAM size ps aux|grep jboss : Process status of jboss jboss-stop : to stop server /etc/init.d/postgresql-9.2.4 restart : to restart postgres jboss-start : to start server reboot : to reboot the server Useful wiki http://intranet.wavelet.asia/projects/tech/wiki/%22What_to_do_when_server_is_down%22-A_complete_write_up http://intranet.wavelet.asia/projects/tech/wiki/Checking_Performance_of_Linux_System_(_using_%E2%80%9Ctop%E2%80%9D_command_) http://intranet.wavelet.asia/projects/tech/wiki/High_Availability__Load_Balancing_and_Replication_for_PostgreSQL http://intranet.wavelet.asia/projects/tech/wiki/JBoss_ http://intranet.wavelet.asia/projects/tech/wiki/JBoss_configurations http://intranet.wavelet.asia/projects/tech/wiki/Connection_Pool_size |
| 9 | Server Database Server | PostgreSQL states: - Check to make sure you are able to connect to the database
- If you can’t, if it is a
- App / DB setup
- Check if network is up (refer to above points 1 to 6)
- Check if database server is up
- If you are not able to connect to the wsemp database, most probably your postgres has not started yet.
- run command as root, /etc/init.d/postgresql-9.2.4 start and try psql command
- Check if PostgreSQL is running
- Check if the database server has rebooted, by using last. If this is occurring not during the cron script time (6am), ALERT FAULTY SERVER FAN!!!!
- Combo setup
- To look at running query
- Analyzing Idle In Transaction,
Postgresql Log Located at /var/lib/pgsql/data/serverlog Errors are usually very apparent, they will tell you it's an error You can locate the related table / query that might be causing the problem
| common error is when postgres cannot start as postmasterpid has invalid data To solve: http://intranet.wavelet.asia/projects/tech/wiki/When_postgres_cannot_start_as_postmasterpid_has_invalid_data |
| 10 | Server Hard Disk | - Run this command
- If the usage is more thatn 90%, you need to clear up some data by finding it using command
- find / -type f -size +20M -exec ls -lh {} \; | awk '{ print $NF ": " $5 }'
- HDD 100% might be caused by /var/log/cups/ . Refer to /wiki/spaces/WU/pages/20579213
Note: Don't delete anything inside /var/lib/pgsql/data, we stored database inside that folder. It's normal for that folder have big size | |
| 11 | Wavelet EMP Itself | Server slowness can be found only from specific module from EMP caused by Inefficient code or SQL queries - Get from customer which module causing slowness and test from your personal PC or Laptop whether the module is really slow. Run command showlog, observe whether there is any exception error found. If you are not confident on the code, get any programmer to sit beside you and help to troubleshoot it.
- Variety of exceptions:
- org.postgresql.util.PSQLException. Solutions: copy the query before that error appeared and report to programming team to fix the SQL query.
.png?version=1&modificationDate=1450851111992&cacheVersion=1&api=v2&width=426&height=250)
- nullpointerExecption. Solutions: Repopulate the database to your localhost. Run the debug or your eclipse and found which table or queries causing null pointer exception. In most cases, error will be solved if you find which table has null data and update it to some values depends on the "Type". Example, if the Type is integer, you can update to 0, If it is string, update to '' or empty string
- When customer complain their reports run very slow, but after you repopulate the database to your localhost no such thing as slowness occurred. What you need to do
- Check the whether the VACUMM is running.
- To check as empbackup users. run vim backup.log and you need to make sure the latest backup log shows the VACUMM is done.
.png?version=1&modificationDate=1450851541846&cacheVersion=1&api=v2)
- Inform customer that you will repopulate database again at night. In same cases, the slowness is gone after we re-populate their database. This can't be done at working hours unless customer ask you to
- Common cases is that Modules is missing from EMP. Please refer to below wiki to fix the problem.
| Check the server log using vim command vim /usr/java/jboss/server/default/log/server.log |
| 12 | Usage of EMP | Heavy usage of EMP from customer : Need to find out which report and who is generating and communicate with customer about the usage of report. The usage of query checking running process would be useful to find out the specific report | To check Queries to check running processes http://intranet.wavelet.asia/projects/tech/wiki/Queries_to_check_running_processes .png?version=1&modificationDate=1450777976387&cacheVersion=1&api=v2)
|