1) Too much contention
You claim to have multi-threaded applications. Multi-threaded model does not mean faster. Your apps will not get the boost from multi-threaded model and multiple CPU/cores if you have too much contention, blocking and waiting. Review your threading model and weed out contention. Excessive contention will reduce your app performance as load increases. Adding more CPUs won’t result in performance improvements.
2) Bad data model
Does your app access data? Is your app write or read heavy? Your app performance will suffer if your data model does not optimize for most common set of use cases. Say you have a read intensive app and you store data in SQL. Improper data model design, bad index choices will kill your application performance. Carefully look at the use cases and scenarios and examine how your data model supports it.
3) Too much synchronous IO
Does your application wait fora database or messaging to get back to the user? If so, consider swapping from synchronous to asynchronous communication model. This way your app can return back to the user quickly and let the processing take its course in the background.
Request completion can be reported back to the user asynchronously much later without imposing a blocking wait and making your users wait.
4) Bad UI/UX design
Yes, your application will be slow simply because it takes too much time to figure out what to do, if your UI is not well designed. Most app developers focus on how long it takes to click, generate report, display a chart, etc. What about how long it takes to figure things out? That is part of your application performance and ultimately affects the quality of service. Focus on UI design and user experience and it will make your application faster.
5) Resources vs Workload Mismatch
Poor capacity planning can literally kill your app no matter how well it is designed: limited memory, heap, not enouph CPU, poor heap/GC model for java applications. We all know what happens with java apps when GC kicks in — hangs, timeouts etc. Most shops throw hardware or VM instances at this problem, which can alleviate the problem to a certain degree. Tuning java GC can be very challenging for server apps with varying workloads. Having a good handle on the application workload vs available resources is critical for optimal application performance.
6) Bottleneck effect
No matter how well your application is designed it will be as slow as your slowest link. Network, DB, application servers — all have to be fast to deliver optimal performance. Pay attention to infrastructure components that follow hub and spoke, sink models. Any hub or a sink can become a performance killer. Typical examples of hubs and sinks are: web servers, brokers, databases, routers, middleware. Relieve hub/spoke dependencies, use redundant, replicated infrastructure, and eliminate bottlenecks by removing single points of failure.
7) Too much linear processing
Sometimes your application just needs a lot of processing that takes a lot of computations on a lot of data. It takes time. Why do it serially?
Attempt to break the problem into smaller pieces and execute your tasks in parallel. Consider clustered computing platforms such as Spark, STORM, MapReduce to execute tasks in parallel across a cluster of servers. Aggregate your data in advance and reduce data processing overhead for most common use cases.
Optimize your data model for how it will be used and not just on how efficiently the data is stored. You will have to compromise somethings to get faster processing in return.
8) Lack of tools and process
Things break, slow down – entropy is everywhere especially in your apps. Do you know when things fail or slow down? Do you have the tools to deal with performance bottlenecks and failures? Adopting a solid methodology such as DevOps will go a long way to ensure your apps are at their best.
There is never too fast. Your apps could always be better, faster, and snappier. Establish a good handshake between end-user, Ops and Dev and watch your app quality improve. Consider Unified Application Analytics platforms that let you consolidate end-user, performance, logs and other application and infrastructure metrics into a single view. Having to jump from tool to tool to get visibility will drive you crazy and add time to MTTR (mean time to repair).
9) Bad developers
Dump developers who don’t pay attention to performance. Today it is easy to write bloated code that works but slow. In java for example, developers who have a good grasp of C/C++ as well as assembler understand performance better than developers who only work with higher level languages. Understanding hardware, OS, memory management and IO is a huge plus, especially when delivering high performing applications.
10) Mismatched performance expectations
You think your application is fast but your end-users think it’s slow. Why? Your application performance is very much dependent on end-user expectations. What is fast for some is very slow for others. For some, a few seconds response time is fast, for others eternity. Compare performance expectations of a high frequency trader vs someone buying a TV on eBay or Amazon. There is a huge difference in performance expectations. Know your user and design accordingly.