Cannot Reproduce a Bug? – DZone

Several tools such as strace, dtrace, and others can provide deep insights into a running application. This information can help us pinpoint differences and misbehaviors within the application. The advent of container technology like Docker has greatly simplified the creation of uniform environments, eliminating many subtle differences.

I was debugging a system that only failed at the customer’s location. It turns out that their network connection was so fast, the round trip to the management server was completed before our local setup code finished its execution. I tracked it down by logging in remotely to their on-site machine and reproducing the issue there. Some problems can only manifest in a specific geographic location.

Ideally, we shouldn’t reach these situations since tests should have the right coverage. However, in practice, this is never the case. Many companies have “long-run” tests designed to run all night and stress the system to the max. They help discover concurrency issues before they even occur in the wild. Failures were often due to lack of storage (filled up everything with logs) but often when we got a failure it was hard to reproduce. Using a loop to re-run the code that failed many times was often a perfect solution. Another valuable tool was the “Force Throw” feature I discussed previously. This allowed us to fail gracefully and pass stumbling blocks in the long run.

Logging

Yes, logging requires forethought much like observability. We can’t debug an existing bug without logging “already in place.” Like many things, it’s never too late to start logging properly and pick up best practices.

If a bug is elusive the odds of a concurrency-related issue are very high. If the issue is inconsistent then this is the place to start, verifying the threads involved and making sure the right threads are doing what you expect.

Use single thread breakpoints to pause only one specific thread and check if there’s a race condition in a specific method. Use tracepoints where possible instead of breakpoints while debugging – blocking hides or changes concurrency-related bugs, which are often the reason for the inconsistency.

Review all threads and try to give each one an “edge” by making the other threads sleep. A concurrency issue might only occur if some conditions are met. We can stumble onto a unique condition using such a technique.

Try to automate the process to get a reproduction. When running into issues like this, we often create a loop that runs a test case hundreds or even thousands of times. We do that by logging and trying to find the problem within the logs.

Notice that if the problem is indeed an issue in concurrent code, the extra logging might impact the result significantly. In one case I stored lists of strings in memory instead of writing them to the log. Then I dumped the complete list after execution finished. Using memory logging for debugging isn’t ideal, but it lets us avoid the overhead of the logger or even direct console output (FYI console output is often slower than loggers due to lack of filtering and no piping).

While it’s never truly recommended to “give up,” there may come a time when you must accept that reproducing the issue consistently on your machine is not feasible. In such situations, we should move on to the next step in the debugging process. This involves making assumptions about the potential causes and creating test cases to reproduce them.

In cases where we cannot resolve the bug, it’s important to add logging and assertions into the code. This way, if the bug resurfaces, we’d have more information to work with.

The adage “it really works on my machine” typically falls quick on the earth of software program growth. We should take possession of bugs, attempting to duplicate the consumer’s atmosphere and behaviors as carefully as attainable. Clear communication is essential, and integration between R&D and assist departments may be invaluable.

Trendy instruments can present deep insights into operating purposes, serving to us to pinpoint issues. Whereas container applied sciences, like Docker, simplify the creation of uniform environments, variations in networking, knowledge sources, and scale can nonetheless impression debugging.

Typically, regardless of our greatest efforts, bugs cannot be persistently reproduced on our machines. In such instances, we have to make educated assumptions about potential causes, create check instances that reproduce these assumptions, and add logging and assertions into the code for future debugging help.

In the long run, debugging is a studying expertise that requires persistence and adaptableness and is essential for the expansion and enchancment of any developer.