Know your Byte Code

This is an amazing write-up:

HikariCP offers a high-performance JDBC Pool

Manage Node Connection Provider

Curator Framework simplifies ZooKeeper :  offers robust management for distributed systems.

Ref :

Automatic connection management:

  • There are potential error cases that require ZooKeeper clients to recreate a connection and/or retry operations. Curator automatically and transparently (mostly) handles these cases.
  • Watches for NodeDataChanged events and calls updateServerList() as needed.
  • Watches are automatically removed by Curator recipes

Cleaner API:

  • simplifies the raw ZooKeeper methods, events, etc.
  • provides a modern, fluent interface

Recipe implementations (see Recipes):

  • Leader election
  • Shared lock
  • Path cache and watcher
  • Distributed Queue
  • Distributed Priority Queue

More specifically ,

The Curator Framework constantly monitors the connection to the ZooKeeper ensemble. Further every operation is wrapped in a retry mechanism. Thus, the following guarantees can be made:

  • Every Curator operation properly waits until the ZooKeeper connection is established
  • Every Curator Framework operation (create, getData, etc.) is guaranteed to manage connection loss and/or session expiration per the currently set retry policy
  • If the connection is temporarily lost, Curator will attempt to retry the operation until it succeeds per the currently set retry policy
  • All Curator recipes attempt to deal with connection issues in an appropriate way

Always Manage Executor Threads:

  • — converts the given ThreadPoolExecutor into an ExecutorService that exits when the application is complete.
  • Example :
    • maintain a safe map of listeners –  private final Map<T, ListenerEntry<T>> listeners = Maps.newConcurrentMap();
    • each listenerEntry caontains – addListener(listener, MoreExecutors.sameThreadExecutor());

Provide a Facade over Queue:

Since a queue can handle only certain number of items, say ZooKeeper’s transport layer ,  a facade like QueueSharder can be used to monitor the queues and if any one of them goes over a threshold, a new queue is added.

Always broadcast Notifications :

ConnectionStateListenable().addListener(new ConnectionStateListener() {
public void stateChanged(CuratorFramework client, ConnectionState newState) {
                System.out.println(“** STATE CHANGED TO : ” + newState);

Provide pluggable Error Policies

Calculate the Metrics

Enqueue streaming data and process streams

example: create an Storm Event Processing topology :

  • generate Spout configs (scheme, offsetTime, buffSize, fetchMaxWait, socketTimeout….)
  • spawn QueueSpout (say Kafka)
  • tighten the input and output Bolts
    • leverage the fact that this Bolt can accept a Tuple and emit collection to OutputCollector.
  • config Storm Nimbus
  • submit Topology to a cluster

Embrace smart messaging proxy

Example :  Fabric8MQ

  1. Protocol Conversion — by moving MQTT,STOMP,AMQP protocol conversion outside of the ActiveMQ broker reduces the amount of work the broker needs to do: As ActiveMQ isn’t well designed for modern, asynchronous tasks, the less work the broker does, the better.
  2. Camel Routing is built right into the router — so we can easily convert on the fly between, say Topics (Publish/Subscribe) and Queues (point-2-point).
  3. API Management — utlizing API Man, so you can apply security and rate limiting policies to destinations
  4. Multiplexer  — reducing the amount of physical connections an ActiveMQ broker has, increases the throughput of the overall system.
  5. Destination Sharding — this is where a lot of the magic pixie dust is used, to shard connections across many different backend ActiveMQ brokers. Its also where messages and clients are moved between brokers, as brokers are spun up and down, depending on load
  6. Broker Control — this monitors the brokers under Fabric8MQ’s control and monitors load — and it decides if more ActiveMQ brokers are needed, or less — depending on load. An embedded rules engine is used to make the decisions.

Always adopt a flexible Test framework:

Adopt a highly efficient and expressive Data Structure:

example :

 excerpts from ..
 6 public Iterable<Integer> findAllIdsForLastNameSortedByIdUsingJava7(
 7         Iterable<Person> people, String lastName) {
 9     return FluentIterable
10             .from(people)
11             .filter(new Predicate<Person>() {
12                 public boolean apply(Person input) {
13                     return input.getLastName().equals(lastName);
14                 }
15             })
16             .transform(new Function<Person, Integer>() {
17                 public Integer apply(Person input) {
18                     return input.getId();
19                 }
20             })
21             .toSortedList(Ordering.natural());
22 }

References :

  • MutableList<Address> addresses = people.collect(Person.TO_ADDRESS);
public class Person {
public static final Function<Person, Address> TO_ADDRESS = person -> person.getAddress();
  • MutableList<Person> adults = -> each.getAge() >= 18);

many more cool high-performance library ..  Agrona  , FastUtil

Simplify Regex Handling:

Finally a Great Library :

VerbalExpression testRegex = VerbalExpression.regex() .startOfLine().then(http).maybe(s) .then(://) .maybe(www.).anythingBut( ) .endOfLine() .build();


Always use an efficient Serializer:

eg: Kryo is a fast and efficient object graph serialization framework …

Track latency of Operations:

Servers should use Non-Blocking-IO

e.g.  Netty , Grizzly

Implement Concurrency best practices

ThreadMXBean threadMXBean  ManagementFactory.getThreadMXBean();

long[] ids = threadMXBean.findDeadlockedThreads();

if (ids != null) {

  ThreadInfo[] infos = threadMXBean.getThreadInfo(ids, true, true);

  for (ThreadInfo threadInfo : infos) {



 System.exit(127);   // shutdown the application



Process process = null;

int exitcode = 0;

while (true) {

process = Runtime.getRuntime().exec(“java -cp . DeadlockedApp”);

exitcode = process.waitFor();

System.out.println(“exit code: ” + exitcode);

 if (exitcode == 0)



  • Instrument a class file using custom agent  , kenai/btrace and don’t be afraid to capture live variables
  • Use HSDB to connect to heap space of any live JVM
  • wherever possible adopt the goodness of CAS  (no locks, no volatile) :
    • no reads for most reads and most puts unless hash is broken.
    • directly reach ByteArray in O(1) .
    • copy array to a larger array on demand (post updates)
  • adopt guava multi-map , peach predicates
  • use Google proto-buffer for serialization
  • use TimeUUIDs , com.eaio.uuid much faster  (java.util.UUID is slower)
  • compile-time dependency injection :
  • create a quick command line tool for Java API  :
  • use for better String performance and try -XX:+UseCompressedStrings
  • always expose critical process-heavy entities as ServerMXBean and emit notifications.  ( Plain Old Java Wisdom)
    • keep track of application states , flush caches
    • Netflix Servo to publish metrics
  • Use @FunctionalInterface to define signature of lambda expressions in an interface
  • some clean Java 8 code :

ref –

List<String> students = range(0, 100).mapToObj(i -> randomString(new Random(), 'A', 'Z', 10)).collect(toList());
// sort names and group by the first letter
Map<Character, List<String>> directory = -> name.charAt(0)));
// print a nicely-formatted student directory
directory.forEach((letter, names) -> System.out.println(letter + "\n\t" +"\n\t"))));

public class RatingCommand extends HystrixObservableCommand<String> {

    private int id;

    public RatingCommand(int id) {

  super(HystrixCommandGroupKey.Factory.asKey(“RatingGroup”)); = id;



    protected Observable<String> construct() {

        return RxNetty.createHttpGet(“http://localhost:9001/&#8221; + id)

                .flatMap(response -> response.getContent())

                .map(data -> data.toString(Charset.defaultCharset()));



    protected Observable<String> resumeWithFallback() {

        return Observable.just(“{\”id\”:” + id +”,\”rating\”:\”3\”}”);



  • use Fiber to fix Thread : consider Fibers for fault-tolerant concurrent systems (create thousands of lighwt fibers at the cost of some OS-level threads)
  • lets not forget basics Java Data Connectivity – and advanced Jdk 8 Streams to group / map / collect data
  • embrace JWT Token Authentication 
  • simple reminders : judicious usage of java 🙂
  • Off-heap Direct Byte Buffers are always very fast
  • always measure performance
    • JVM analysis :  dtrace , hprof , introdpec , visualvm , yourkit , gchisto
    • OS tools : dtrace , oprofile , vtune , perf
    • N/W , Disk analysis :  ganglia , iostat , lsof , netstat , tcpdump
      • measure cpu performance using ‘Mixed mode stacks’ , ‘Aggregate CPU cache misses’ !
      • basic conf :  java -agentlib:perfagent  -Xcomp
  • advanced profiling : sudo perf record -e hotspot_debug:symtab -e hotspot:object alloc java -cp  /app/code/test -XX:+ExtendedDTraceProbes -XX:DTraceAllocProbes helloworld
  • sudo perf report –show-total-period
  • advanced profiling with custom visualizations :  heapster with graphana dashboards
  • it never hurts to recap memory allocation :  [OS + C Runtime, JVM + Native Heap,  Java Heap]
    • java heap  [Runnable , ThreadGroup, Thread] => native heap [Native Thread, Thread Stack, Java Stack]
    • let GC occupy N% empty heap so that GC does not occupy the entire CPU  (N should be tuned as per your server)
    • tool :
  • useful metrics to tune GC
  • -Xmn for predictive performance (where lots of new objects are created and die in young)
  • Avoid CMS heuristic (adaptive) for compaction :
  • -XX: +UseCMSInitiationOccupancyOnly
  • multi-core GC setting : -XX:ParallelGCThreads=4
  • avoid OOM by setting  -XX:MaxDirectMemorySize=4G
  • Heap Population (How much alive ? )
  • Allocation Rate (How fas allocated ? )
  • Mutation Rate (How fast program updates references ?)
  • Heap Shape (live object graph complexity)
  • Object lifetime
  • Cycle time (time taken by collector to free up memory)
  • Marking time (how long it takes to find all live objects ?)

    -verbose:gc (print the GC logs)
    -Xloggc: (for more comprehensive GC logging)
    -XX:+PrintGCDetails (for more detailed output)
    -XX:+PrintTenuringDistribution (displays the tenuring thresholds assumed by the JVM)

  • don’t forget to use gradle , docker , mesos for modern java apps :
  • finally Project Apex is here as the fault-tolerant distributed system using java :

References :