Should Wolfram Mathematica consider a public bug tracker?

Wolfram Research, Inc. (WRI), the company behind Mathematica, one of the largest computer algebra systems, has as long as I can remember email-based user support. While this is reasonable for support requests where people seek help using the software, it is a painstakingly inefficient way for handling bug reports from professional Mathematica users. One of the main reasons is that there is no publicly available list of known issues and if you find a bug, you have no way to search if it was already reported or if developers are already working on a solution. The email-based approach has further downsides compared to a public issue tracking system we know from, e.g. GitHub, YouTrack or FogBugz:

• Comments from WRI developers or other users, the status of a bug-fix, related issues, or even workarounds from other users are not available to you. All the information you have is the discussion between you and the support.
• You don’t have a single place where all the information about an issue is collected as you have on GitHub. Your bug-report is spread through several emails and code for reproducing a problem is usually located in notebooks somewhere else on your computer.
• Everything depends on so-called “case numbers” that are assigned by WRI when you report bugs. Since you don’t have that case number when you first send the report, you might not even be able to find the exact email you sent in the first place. Keeping track of an issue with your email client is clumsy compared to the alternatives that are available nowadays.
• While on issue trackers the developers themselves comment, the communication with Wolfram always goes through a support employee as a man in the middle who then reaches out to the developers and reports back. This is not only more time-consuming, but it has happened that you need to convince the support than an issue is indeed a deficiency.

For me as a developer that means I have to decide if I indeed want to spend my time writing a concise report and minimise the code necessary to reproduce the issue. Since I cannot be sure that other people haven’t already reported the same problem, I usually work around the bug without bothering the support.

I have talked to other people having similar opinions. It is essential to understand that we do care about the quality of Mathematica and when we report issues, we want to know if it was confirmed as a bug, if there is a report for it, if it’s being worked one, etc. Currently, this is not possible. It also happened that WRI support could not find mention of a problem even though it was reported earlier. Several people on StackExchange expressed the opinion that writing to support sometimes feels like shouting into a black hole. If there was a central place for all known issues and making it available to Mathematica users would build a lot of trust with the community.

Other companies already handle their issues in the open, giving the community the opportunity to participate and see the status of a bug. Below is an example from the bug database of IntelliJ IDEA showing all details about a deficiency including information like who is the assigned developer, is it indeed a bug, is there a goal for fixing it or are there related issues. Additionally, all registered users can comment, provide further information or vote for issues.

The question is why hasn’t WRI considered using an issue tracker which is also available to registered Mathematica users when other companies like JetBrains or Google have already shown that it is beneficial to maintain a publicly accessible bug database? This is hard to answer, and we should not make any presumptions about WRI’s motives. However, I believe two questions are worth thinking about:

• Will the implementation of a public issue tracker increase the workload of support and developers compared to the current approach?
• Will an open bug list harm the reputation of the company?

The first point is indeed hard to answer without knowing the current work-flow. Assuming that WRI already maintains an internal bug database, one could argue that the overall workload doesn’t change much, but the responsibilities shift. Instead of communicating over email, support employees would be assigned to maintain the bug tracker by moderating and tagging issues. WRI developers would use the public database instead of the internal one. I’m aware that this might be a vast simplification of the actual work necessary to make this transition.

Fearing a loss of reputation is a crucial point which is, in my opinion, not entirely unfounded. There is a clear difference between a bug in, e.g. the UI of a browser and a bug in a computer algebra system used in many scientific projects where results need to be reliable. Some of you might remember the report about a deficiency in Mathematica with matrices of large integers which went viral a few years back. There were quite some people that took this opportunity to brag along the lines: “Told you not to trust closed software.”

However, open software is not free of bugs either and one of the major points of people is that they at least have the choice to look at the code and fix the bug. Many tend to forget that only a small minority would even be able to fix such an issue in a software with a code-base as large as that of Mathematica. However, the critical point is that openly known deficiencies are far better than users mistrusting black-box software with unknown bugs. If you read the conclusion of the report above, you’ll find that the authors have a similar opinion:

Having made this criticism, let us stress that software systems have proved very useful to research mathematicians. […] Software bugs should not prevent us from continuing this mutually beneficial relationship in the future.

Therefore, it is safe to assume that most users of Mathematica would also agree that with an open bug database the benefits would exceed disadvantages. I want to go one step further and speculate that many users would trust Mathematica more knowing that WRI is not secretive about bugs. On a side note: Did you know that IntelliJ IDEA has currently about 6000 open issues? Me neither! But what I do know is that it is one of the best software I ever used. See? It’s that simple.

To not entirely rely on my personal opinion for this post, I tried to research if someone smarter than me has already said something about “why a company which sells software should use an open bug tracker”. I wasn’t able to find something substantial through Google, so I reached out to a friend of mine at JetBrains asking how they knew their open bug tracker would work. It turns out, there doesn’t seem to be a secret whitepaper about their reasoning, but one thing he jokingly said struck me:

it works, because we have smart users

But what exactly is a smart user? One point is that people using IntelliJ IDEA are developers themselves and not typical end-users. However, I claim you don’t need to be a full-stack developer to possess the important features of a smart user. People who are good at tracking bugs down and helping to make software better often share some of these characteristics:

• They know when they found a bug opposed to using the software wrong.
• They can communicate issues accurately, create minimal examples, and investigate ways to reproduce faulty behaviour.
• They can often even explain, what goes wrong and how a bug can be resolved.
• They are smart enough to understand that harmless-looking bugs might take substantial time to fix.
• They are often solution-oriented. They know that bugs happen all the time and that there is no need for drama when something doesn’t work.

Do these characteristics apply to the typical Mathematica user? I am not entirely sure, and it is a bit dangerous to argue that Mathematica is a developer-focused product. The feedback we heard more than once from WRI is that for the vast majority of people it is enough to use the features of Mathematica and that package-developers are only a tiny fraction of Wolfram’s customers. However, people like Szabolcs Horvát who develops the IGraphM package are highly skilled with the Wolfram Language and have in-depth domain knowledge in their respective field. These are the people who dig deep enough to stumble over issues, write excellent bug-reports and help to make Mathematica better.

My last point is that it is crucial to understand that an open bug tracker is not an all-in or nothing step. First, bug tracking software allows for private issues which makes it easy to hide reports containing internal implementation details. Second, it is not necessary that everyone can create or comment on issues. The goal should be to prevent that the bug tracker is used as a support platform. So it might not be the best choice to make it openly editable to everyone. However, there are certainly scenarios that would work well for a Wolfram bug database:

• Registered Mathematica users can search the bug tracker and look-up existing deficiencies.
• Certain or all registered Mathematica users can vote for issues and possibly provide further information like the operating system or minimal examples to reproduce the problem.
• Trusted users can also create issues and have more possibilities to interact with the developers directly.

I want to believe that such an open bug tracker for Mathematica nurtures the trust of users instead of driving them away and there are clear mutual benefits for users and WRI

• When reporting and commenting on issues is as easy as on GitHub, more people will participate.
• Things like voting for issues helps WRI to get a better picture of which problems affect most people.
• Experienced users might even look into issues themselves and track down the exact circumstances that lead to faulty behaviour. That might take some work off of the developers.

An open system for bugs is, in my opinion, only the next logical step in a series of actions WRI took to connect to its user-base. For years, WRI employees are a crucial part of our community on StackExchange, and we highly value their contributions. With the Live CEOing Twitch streams, Stephen Wolfram started to give users a look behind the curtains and went so far to discuss most features of Mathematica 12 long before the release. In the pre-release period of version 12, WRI initiated a chat-platform where pre-release testers could directly discuss issues with WRI developers. Was any of this harmful to Mathematica’s success? I highly doubt it. If anything, it has brought the community closer to the creators of their beloved computer algebra system.

Acknowledgements

Although, I discussed this topic with several people over the last years, I’m particularly grateful to Szabolcs Horvát who commented heavily on a first draft of this post and I tried to incorporate most of his excellent arguments.