Java tracker thread shutdown

For engineers Tracking SDKs
1

Mostly, we’ve been using the Javascript tracker, which has worked very well. Now we are trying to implement tracking in a Java app.

Following the instructions at https://github.com/snowplow/snowplow/wiki/Java-Tracker, everything works as expected. However, the Java app hangs rather than exiting (the app reaches the end of main() rather than calling System.exit()).

It seems that the threads from the ExecutorService from the BatchEmitter class are preventing the exit (I am not a Java expert). My workaround is to change the close() method (as in the Android tracker).

  public void close() {
    flushBuffer();
    if (executor != null) {
      executor.shutdown();
    }
  }

Is there any other alternative that doesn’t involve changing the imported classes?

3

Hey @Brian - does it hang indefinitely or does it eventually close? The Tracker attempts to do a final flush which might not be exiting cleanly…

I have created a ticket to track this issue so we can get it resolved: https://github.com/snowplow/snowplow-java-tracker/issues/187

Your workaround looks like what needs to go into the Tracker to - as you mentioned - mimic the behaviour of the Android Tracker. A PR into the tracker repository would be most welcome!

4

Thanks for the comment. No, it never closes. The issue is that the threads don’t die when the end of main() is reached. They only die when killed manually or by System.exit().

There is an issue with my proposed fix; because flushBuffer() call does not block, it’s possible that the subsequent call to executor.shutdown() is made before the events are sent.

When I created an app sending lots of events this caused some events not to be sent and the HttpClientAdapter to raise plenty of exceptions (we closed the HttpClientAdapter as soon as the call to close() returned).

Anyway, i think the easiest solution is to have a time-out on the shutdown (alternatively, we could keep track of all events buffered and wait until they are all sent).

import java.util.concurrent.TimeUnit;
...
private long closeTimeout = 5;
...
public void close() {
    flushBuffer();
    if (executor != null) {
        executor.shutdown();
        try {
            boolean e = executor.awaitTermination(closeTimeout, TimeUnit.SECONDS);
            if (e == false) {
                LOGGER.debug("Executor service shut down after time out");
            }
        } catch (InterruptedException e) {
        }
        executor = null;
    }
}

Am happy to create a PR with this.

5

I was affected by this problem as well. In my code, I used explicit System.exit() and as the result no snowplow events were ever sent out. To combat the issue, I established an java.util.concurrent.atomic.AtomicInteger singleton counter. The counter is incremented every time an event is tracked and decremented by the number of success + fail event counters in via emitter callback. Prior to exit I sleep in 300 ms. increments, flushing emitter buffer and waiting for the atomic counter to get to 0.

A. This solution is not taking into account any retry / backoff strategy that may be in play.
B. It also seems that if the event generating thread is busy the event emitter threads never get to execute (even when I Thread.yield() ). So the bulk of the submitted events is sitting waiting to be emitted at the end of the program execution.
C. I think a better solution would use a shutdown hook [ Runtime.getRuntime().addShutdownHook(Thread) ] to hide the particulars under the hood, just like deletion of temporary files on jvm exit is handled in the File module.
A more comprehensive resolution would be a huge welcome. Unfortunately I’m not that good at concurrency to offer a PR.

1 Like