Bug Hunting

When you come across a bug in an application, what's your reaction? Annoyance? Satisfaction? Something else?

For the past several months, I have been helping a client implement a new Enterprise Resource (ERP) system. My role was to assist the client's Operations folks and make sure everything in the base system and a Production-related add-on module worked well together. Because of a series of unexpected events, recently I was asked to take over the day-to-day project leadership role, while also continuing my "deep dive" into the Operations and Production functionality.

As a result, ERP systems - and software systems in general - have been top of mind this summer.

When I have to get up to speed on a particular system, I do so by trying out the system and making liberal use of the Help options and any available Training Guides. My approach is to understand the philosophy of how a system works, and to do this as quickly as possible. Thus, my goal is to find bugs – go “bug hunting” - and uncover "gotchas" as quickly as possible.

Sometimes, system developers appreciate my approach - sometimes, they don't. Regardless, I do what's in the best interest of my client, who is always very glad I've identified concerns sooner rather than later.

What I'd like to share today are some best practices from application development efforts I've observed, been part of, or done myself. Keep in mind, I do know a thing or two about application development. I'm particularly proud of one workflow application where I personally designed and developed the server-based workflow routing functionality, and how my code has NEVER broken or had to be altered in over 15 years of continuous operation.

Please note that every software system has its share of great, good, and so-so features. And bugs. I frequently refer to bugs as "undocumented features." Thus, even though some of what I'm sharing was triggered by my recent experiences, the following comments come from my 35+ years of experience being involved with systems designed to achieve business results.

Software Development Process - An Overview

To set the stage, here is how I view the general software development process:

• Functional Design - This first stage is where a business analyst works with end users to determine both the "current state" and "future state" business process, and to define how users will interact - both entering information and obtaining results - with the system. I refer to the interactions between users and the system as the "conversation flow."
• Technical Design - After the business analyst completes and documents her work, she works with a systems analyst. The systems analyst relies on the knowledge of, and documentation produced by, the business analyst to produce technical specifications for the system.
• Application Development - The systems analyst goes over technical specifications with an application developer, who uses a variety of techniques, including flowcharts and pseudo-code (English-like instructions of what the application needs to do), to plan development efforts. After the plan is set, the developer begins work, testing his code along the way.
• End-User Feedback - Ideally, the developer will develop functionality in chunks, so that a chunk can be demonstrated to end users for review and feedback. Based on the feedback, the developer will revise the code for the first chunk, check the revised functionality with end users, and use the feedback to guide development of the remaining chunks.
• Acceptance Testing - Whether this is done for a specific function ("unit acceptance testing") or the application as a whole ("system acceptance testing"), end users test the developer's work to ensure it provides the needed functionality.
• Rollout, Shakedown, and Support - Once everyone believes the system is ready to deploy, it is rolled out. For a period of time, users will inevitably find bugs - generally resulting from odd situations never considered or data outside usual limits. As soon as possible, these bugs will be logged, fixed, tested, and deployed. At some point, folks will be confident that the most important issues have been addressed, and the system will be turned over for support.

With the stage set, I can now share practices which yield robust software.

Tips for Developing Robust Software

When I find bugs in software turned over for support, it is usually because of one or more weak links in the development process.

General

• Even though ERP software is very complex and requires large teams of people to develop it, there needs to be a single individual or a single set of standards to force a unified architecture and clear direction of the system, as well as consistent user interface standards - otherwise, end users could become confused and frustrated.
• When being trained to use a new ERP system, I deliberately work AHEAD to the next lesson, and see how naturally the upcoming material flows from what you've learned thus far.
• For commercial software, don't let Marketing and Sales get ahead of the actual features and capabilities the system will deliver!

Functional Design

• Depth beats breadth – that is, I'd rather have a fewer number of very versatile and well thought-out options, versus a bunch of options poorly fleshed-out, scattered about the system, and somewhat haphazard in naming and operation.
• Define terms precisely, and then use them consistently.
• Use flowcharts liberally to map out the overall process flow.
• Use outlines or flowcharts to plan "conversations" between the user and the system.

Technical Design

• Use Venn diagrams to understand data points Situations A and B have in common and where they differ.
• Use tables to enumerate all possible combinations of data points or ranges of data points.
• Use truth tables to list the conditions testing data to distinguish, say, Situations A and B. Conditional tests typically use AND and OR operations, and frequently nest these to determine the final result. Listing all the "True" and "False" combinations for EVERY data point or data point range allows the developer to see patterns and develop accordingly.

Application Development

• Ensure the user can do ONLY what you expect the user to do and ONLY in the situations specified in the design.
• Always allow for, and gracefully handle, unplanned events. The best system I've seen do this will - when an error occurs - automatically collect relevant data, log the error and data in the user's system, send it to BOTH the user's Tech Support team AND the developer's Tech Support team, and display a screen saying an error has occurred and it has already been logged and reported!
• Develop a consistent user interface.
• Follow STANDARD shortcut and user interface conventions – for example, one ERP system I know uses Ctrl+C to Change something. To my knowledge, Ctrl+C has ALWAYS been the Copy shortcut and using this for Change was NEVER a thing!
• Provide drill down capabilities pretty much everywhere. If you have a grid of information in an Excel-like format, users have become sufficiently familiar with Excel's Pivot Tables to know double-clicking a cell in the summarized grid will open the supporting details.
• Do your work with the assumption that what you're developing will be in use - untouched! - for at least the next 10 years.
• Understand your users WILL include someone like me, who likes to throw odd situations at software and see whether or not it handles it. Developer beware!

Testing and Deployment

• When you find a bug, celebrate your success briefly, and then report it, along with as many details as you can, to help the developers reproduce the issue and fix it.
• Stress test your software. We are working with a client where a software development company went from prototype to deployment without running a typical volume of transactions through the application. The result? A total disaster!
• Provide both adequate Help documentation, and on-screen pop-up help specific to the field or dialog box.
• In an ERP system, searching the Help topics or the online Knowledge Base (KB) should follow the same search conventions as Google. These days, a simple "ALL words/ ANY words" search option is only a starting point - more advanced search options need to be available. For example, I want to be able to search for all Help or KB articles by:
  • An exact phrase - "workflow services"
  • An exact phase and a specific word - "workflow services" trigger
• You can test variations on these searches in Google and see the results. Note that, in the second example, Google places an implicit AND between the two search terms that, "workflow services" AND trigger.

Test, Test, TEST!!

• I cannot stress this enough – testing should occur at every point in the process and always be very thorough!

One final note - the best development and testing cannot overcome a marginal functional or technical design. For example, I believe Microsoft Excel has a much better functional and technical design than Microsoft Word. At least for me, Word tries to anticipate what I want to do - and it inevitably gets things wrong. In contrast, Excel almost always helps me the way I like.

How to Be a GREAT and KNOWLEDGEABLE End User!

After reading this article, you are now equipped to be ...

• A GREAT end user - to me, this means REALLY testing a system and then QUICKLY reporting, in as much detail as you can, what you did that led to the bug.
• A KNOWLEDGEABLE end user - equipped with an overview of the software development process, you likely can now pinpoint the weak link that allowed this bug to occur.

And if you need an experienced guide for your "bug hunting"? Give me a call and we'll plan your hunting expedition!

Sincerely,

Todd L. Herman