On Target Testing

Published
by
jw
on December 17, 2008
in Uncategorized
. 0 Comments

On-Target Testing

Object Oriented Design (OOD) concepts have been used for years to develop software that is readily testable and maintainable. The OOD approach models a system as a collection of cooperating objects which are instances of a class within a class hierarchy. Here at SysTest, we have adapted these concepts to develop test methods which are a modular set of test objects within a class. Test methods verify a system’s compliance with a tightly coupled sub-set of system requirements. By linking individual test methods into test suites we derive a cohesive flow used for product specific certification testing that is repeatable, has clarity of requirements coverage and is economical in application; even for complex products such as voting systems.

 

Repeatability and Transparency

Implementing logical groupings of requirements within each method provides inherent traceability between system requirements, the test methods, and the suites of test modules that are implemented. This supports the determination of if, how and where a requirement was met. The object oriented testing approach also provides consistency among all tests. The test modules are populated with actual input data that remains static. Thus, when these modules are incorporated into suites, they carry their data sets with them, leading to consistency of data.

Getting it Done

We have observed that object oriented test methods enable efficiencies in initial product assurance as well as during the inevitable re-testing that is needed as modifications and enhancements are included. This relatively simple solution can deliver real value, especially in larger and more complex IT systems and products. Goals that commonly include schedule milestones, quality metrics and cost of development must apply to the testing portion of any project if delivery targets are to be met. Targeting efficiencies in test methods from the start is a good way to give your project it’s best chance for success in what is often an underestimated cost area.

The material on this page is Confidential and Proprietary to SysTest Labs

Risky business: When companies make quality assurance an afterthought

Published
by
admin
on October 20, 2008
in Uncategorized
. 0 Comments

Risk is inherent in any endeavor to develop or integrate a new product. This is especially true in the software industry, where risk is directly proportional to the level of complexity or size of the system.

As a 27-year veteran of the software quality assurance profession, I’ve made it my business to understand the complexities of software development across a range of industries in order to gauge the level of risk associated with developing new software or integrating existing components. I’ve done this primarily to support my company, SysTest Labs, but I also maintain a professional interest in understanding the industry and the latest trends in mitigating risk associated with software development and integration.

The “criticality” question

Assessing risk requires examination of several key questions regarding budget, timing and staffing. The answers to these questions form the basis of the initial risk assessment.

However, there is one question that must be answered before any risk assessment can begin.

How important is this software?

Is it life-critical, “bet the company” or mission-critical? Can it be implemented enterprise-wide, is it a support mechanism or is it just nice to have? In other words: What happens if this software does not work correctly when deployed?

Let’s call this the “criticality” of the software. If criticality is high, mitigating any risk is vital. If criticality is low, then more than likely, basic business decisions will drive the need to mitigate risks (e.g., budget, available resources and time).

For a proposed system where the criticality is high, an organization should examine its approach to quality assurance and testing, as well as staff experience, subject matter expertise, time and budget.

Early bird catches the bug

Although there have been great efforts to advance the methodologies and techniques used for software quality assurance, most organizations do not implement QA early enough in the lifecycle. The longer an organization waits on QA, the more likely it is that an in-house test team will be the last gate in ensuring a software system meets the expectations of the customer or end user. It’s risky business.

Fortunately, there are several avenues available for organizations seeking to build quality assurance and testing capabilities:

  • Hire a full-time internal team of dedicated professionals
  • Hire subject matter experts and administrative staff
  • Contract individuals or staffing firms for as-needed help
  • Outsource quality assurance and testing responsibilities

Benefits and shortcomings accompany each option. And even though SysTest Labs provides contract and outsourced solutions for quality assurance and software testing, it would be disingenuous to imply that our solution is the best for all situations. Each organization must individually determine its best approach.

Quality assurance and testing teams are often viewed as an afterthought in software development efforts. But because of what they do, when they do it and the impact on your organization if they perform poorly, these teams are vital to reduce risk.

Brian Phillips, President and CEO

Elections across the globe

Published
by
admin
on October 8, 2008
in Uncategorized
. 0 Comments

Last month, I had the distinct honor of providing the keynote address at the 17th Association of European Election Officials general assembly meeting. The meeting was held in Constanta, Romania, a city of great tradition and history located on the Black Sea.

Diplomats and election commissioners from every corner of the globe attended the event. Armenians and Russians, Britons and Mexicans, Brazilians and South Africans. A multi-lingual, multi-cultural conversation on elections. It was quite an experience.

Presentations spanned the gamut of election issues, from security to accuracy of the count to fraud detection and prevention. Even voter safety and protection in the polling place received a good bit of attention.

I focused my address on electronic voting and the role of certification testing in ensuring election integrity. Here in the United States, a very rigorous program of electronic voting machine certification is in place, governed by the Election Assistance Commission (EAC). Election devices must pass a series of tests that ensure adherence to exacting standards. This program provides not just technical validation but confidence to the electorate, that votes are counted as cast and that results accurately reflect the will of the people.

The questions I received surprised me at first. In many nations, election conduct must be policed heavily and fraud and abuse are the norm. Electronic elections are regarded with suspicion for their apparent vulnerability to manipulation. Objection to automation in elections is even greater in many western European nations than some emerging democracies because of the organization and sophistication of the political groups that oppose their use. I came away convinced that the governance framework of the United States - along with extensive testing - is the key to delivering the cost, logistic and social benefits of electronic voting to areas of the world that need it most.

Listening to the other presentations also reinforced my belief that the right to vote, so easily taken for granted here, is a precious gift that many throughout the world continue to give their lives to defend. Being among these passionate defenders of democracy gave me a perspective on the work we do here at SysTest Labs that I will never forget.

Mark Phillips, Vice President of Compliance Services

Backshoring: An emerging cost savings trend

Published
by
admin
on September 8, 2008
in Uncategorized
. 0 Comments

For more than a decade, “outsourcing” has been considered a four-letter word in the United States. Politicians and pundits criticize every industry - from information technology to telecommunications to tennis shoes - for shipping jobs overseas while unemployment rises at home. Yet many corporations still outsource jobs to places like India, where the U.S. dollar carries more weight.

Now, as unemployment reaches a six-year high, a new trend is emerging.

According to a study in the Feb. 4, 2008 issue of Financial Week, as many as 75 percent of 200 companies polled had “backshored” or brought work back in-house. Because it’s difficult to measure how many jobs are being outsourced overseas, it’s hard to know how many are returning, the study noted. However, it is apparent that a significant number of outsourcing dollars are coming back to the States.

The study cited five reasons for the shift:

  • cultural misunderstandings
  • unexpected management needs
  • high turnover in the offshore workforce
  • low skill levels and work that’s too complex

Each of these issues has the potential to quickly wipe out any cost savings on a project derived from what appear to be lower up-front hourly rates.

Michael Fields, former president of Oracle USA, coined the term “backshoring.” And in a May 2006 Fortune magazine article, he predicted what Financial Week would eventually find in its study.

Software development is a collaborative process that works best when everybody involved - designers, programmers, project managers - is together under one roof,” Fields said. “If your team isn’t closely bonded, you’ll see more rewrites, more performance issues and more delays. For us, having the designers and architects in California and the programmers in India has actually meant longer delivery times and higher costs.”

Other factors impact the economics of outsourcing. The falling dollar has begun to cut into the margins between off-shore and domestic project costs. Upward wage pressures in many developing nations increase the cost of labor, reducing the perceived cost advantage. There also seems to be a growing realization that sometimes “you get what you pay for.”

As with any business trend, outsourcing has its place and will continue to be a service solution for certain tasks. But more companies are realizing that a strong domestic service partner can still provide value and return on investment.

The U.S. may very well become the next emerging market as backshoring takes hold.

Jeff Messer
Director of Enterprise Solutions

SysTest Labs supports EAC and NIST/NVLAP governance efforts

Published
by
admin
on September 2, 2008
in voting system testing
. 0 Comments

We have received a number of questions from clients, business partners and media regarding our response to recent clarification requests from the Election Assistance Commission (EAC) and National Institute of Standards and Technology under the auspices of the National Voluntary Laboratory Accreditation Program (NIST/NVLAP).

At the news section of our website, www.systest.com, we have posted the correspondence between SysTest Labs and these organizations to provide information regarding our response. Here, I want to take to opportunity to give further clarification of our view.

Electronic voting holds terrific promise to improve participation in our democracy by providing greater access to all citizens, enhanced convenience, and efficient and timely results. These potential benefits are clear. However, advocacy groups in the US and abroad (particularly France, Netherlands and Ireland) have questioned the transparency, accuracy, and suitability of electronic voting machines.

The EAC has developed an extensive testing program that directly addresses these issues, and SysTest Labs is proud to be one of the very few Voting System Test Labs (VSTLs) accredited to conduct these comprehensive tests on behalf of the EAC.

A key feature of the EAC testing program involves periodic reviews and audits by NIST/NVLAP to ensure adherence to rigorous technical specifications and to verify that the highest standards of professional conduct are followed. In addition to regular periodic reviews, NIST/NVLAP can and must follow-up on any information, regardless of source, that questions the details of program conduct at any time.

This oversight is critical and provides EAC and the voting public assurance that voting machines are thoroughly tested to meet all standards of performance.

We at SysTest Labs recognize the critical role played by the EAC and NIST/NVLAP and are delivering all requested materials and more to ensure these agencies can carry out their oversight tasks as effectively as possible. We understand that by assisting EAC and NIST/NVLAP to the best of our ability, the electronic voting system test program and our contributions to its success will only grow stronger.

-Mark Phillips, VP Compliance Services

Come on in, the wireless door is open

Published
by
admin
on August 20, 2008
in Uncategorized
. 0 Comments

To make sure your home or business is secure, it goes without saying that to keep the bad guys out you must close and lock your door when you’re away. But in our always-on Internet era, why are so many doors to the wireless networks within homes and businesses left wide open?

Though it should be common knowledge that wireless gateways require a locking password, many of these wireless access points (WAPs) remain unprotected.

Just ask your kids. While my internet connection was down over the weekend, my son was still able to access his Facebook account through an unlocked WAP my neighbor kindly provided.

And from my downtown office, I’m able to detect more than 100 WAPs, 50 percent of which are unlocked or feature an unsecure lock to keep their network safe.

Companies need to do a better job protecting their data. And with more and more professionals telecommuting from home, it’s essential they pass this message on to their employees.

First, password protect your network. Then consider a broader security assessment conducted by third-party experts to evaluate the security of your network and key applications and help keep your company and your customers’ information safe.

-Ron Wood, CISSP

ION Games Conference

Published
by
admin
on June 11, 2008
in Uncategorized
. 0 Comments

It was great to attend the ION Games Conference in Seattle. There is quite a buzz brewing about the MMO (online games) segment of the games industry. There are lots of titles in development, the success of World of Warcraft by Blizzard Entertainment, Sony Online Entertainment announced a renewed effort on this front and it seems that everyone is developing IP for this segment of the industry.

PricewaterhouseCoopers has a current forecast report on the industry and says that online games are forecast to be the fastest growing segment of the industry through 2011 growing at over 19% while the console segment is projected to grow about 4.5%. The report also states that “The online market will be spurred by the introduction of new consoles that focus on Internet capabilities enabling players to compete with each other around the world. The growing penetration of broadband will also spur the growth of online gaming.”

PWC refers repeatedly to the expansion of “digital distribution” of games and gameplay. This becomes apparent when you look at what all the console manufacturers are doing. Starting with XBOX Live, we can all get tons of content served up right through the console. TV shows, movies, music and, of course, games. Nintendo has just launched Wii Ware to do the same and Sony has been working on their version PlayStation Home for sometime now with a launch planned for fall of 2008. It would seem that in the near future content of all kinds will come flowing forth in ever expanding streams through our game consoles and to our TVs. Added to the mix on all this should be all kinds of games that we can play with our friends all over the planet. Cool prospect.

Obviously the success of WOW with $1 billion in annual revenue is very appealing to anyone developing IP. Blizzard Entertainment certainly seems to have found the secret recipe for developing revenue generating IP in the online space. Others are holding them up as the poster child for MMOs. NetDevil among others is promising to follow their methodology of game development and their focus on creating a quality product at almost any cost. Blizzard talks openly of never hitting a release date choosing instead to focus on their “gameplay first”philosophy. It does seem to be the secret sauce (at least in part) for online IP revenues. They also talk about cancelling titles that are going well enough. Seem ludicrous for an industry that seems bent on following the movie industry model of releasing a product no matter what due to all the money invested. Turbine is talking some interesting stuff about porting to consoles in the future- http://www.massively.com/2008/06/11/turbine-still-playing-cagey-over-ddo-console-version/

Based on this trend in the industry, SysTest Labs plans to redouble our efforts to provide support services to the “gameplay first” koolaid drinkers. Beyond our QA and testing services we provide PMP certified project management and Agile development consulting (XP, Scrum, etc.). We are also expanding our testing facilities by about 15,000 square feet to ensure we have the space and scalability for multiplayer testing, compatibility testing, compliance testing and customer service support services.

The burgeoning segment of MMO and other online gaming seems like a logical and very cool direction for the industry. We intend to be there to support the industry as it expands into this realm.

Last Day at GDC 2008

Published
by
admin
on February 25, 2008
in gdc 2008 and software testing
. 1 Comment

On the last day, there was a discussion of success stories. Some of this repeats the earlier topics but is worth repeating based on proven successes. I especially took this info to heart in the services we provide our clients.

  1. Add QA and testing to the team early in production.
  2. QA review of design documents.
  3. Prepare for and anticipate problems, don’t get surprised by them.
  4. Use the test documentation like TRCs and TCRs to carefully track and document bugs.
  5. Use tools for effective communication.
  6. Standardize the jargon or vocabulary to facilitate effective communication between teams.
  7. Write test plans and test documents (this is where we shine).

To summarize, there was great information and I learned a lot about the state of QA and testing in the industry. The good news was we are already doing virtually all of this and working to improve these efforts for our clients. It served to reinforce that we are on the right path.

But also, we need to constantly connect with our clients and potential clients and the industry as a whole to stay up on the hot topics to provide the best service we can. I am looking forward to the challenge.

Key Takeaways from GDC 2008

Published
by
admin
on February 24, 2008
in gdc 2008
. 0 Comments

There was a great mix of people at GDC 2008 to discuss the issues of game QA and testing. It seemed apparent that there are no industry standards for QA and testing and the industry as a whole is trying to figure that out.

Here are some of the key takeaways from the sessions I attended:

  1. Start testing early in the development process.
  2. The Agile approach to game development and testing is gaining popularity and focuses on embedding testing into the development process form an early stage.
  3. The skill sets for testers is becoming more sophisticated and having lots of experience or experience in a certain area (i.e. audio) is becoming more and more necessary.
  4. Testing is often done too late in the development process and is done hurriedly and superficially.
  5. If testing starts early, development costs can be reduced.
  6. A major problem identified through out the roundtables is communication between the test team (in-house or outsourced) and the development team.
  7. Documentation: design docs may always be lacking in the ideal detail and specificity. But, details testing documentation (types of bugs, severity, where to find , how to replicate) are crucial to the communication with the development team.
  8. Document storage and version control is important and the group identified Microsoft Sharepoint and Mercury Testdirector as to valuable tools.
  9. The proper use of test cases was also a popular topic.
  10. Drive quality upstream. Back to finding bugs and defects as early as possible even in the design phase.
  11. Set standards, and expectation as to the role(s) and responsibilities of the test team relative to the other functional groups.
  12. Agree on terminology to improve communication between various teams.
  13. Agree on a methodology ie levels of priority and severity for fixes.
  14. Standardize bug reports.
  15. Keep the lines of communications open and work to maintain. This is not necessarily very easy especially at crunch time but it can greatly improve the process.

    Game Developers Conference 2008

    Published
    by
    admin
    on February 23, 2008
    in gdc 2008
    . 0 Comments

    I recently went to my first Game Developers Conference. Though I have lots of friends in the biz, it was quite a sight at Moscone Center in San Francisco as the industry came together for the largest gathering of the year.

    I went to take in everything I could about the current state of the industry, to make as many friends as possible and attend all of the QA/testing roundtables. It was obvious that the industry is very healthy and that there are a great number of exciting developments in the technology, the tools and the functionality of games to come in the near future. Wii Fit was a big hit as Nintendo expands their franchise into new and fun interactive ways.

    There were a couple tracks focus on QA and testing and I was especially interested in these in order for our testing and QA services to be focused on the most current issues that the game development industry faces. Three roundtable sessions over three days (Wed, Thur, Fri) were good coverage of the hot topics. The title of the QA sessions was “What do you need to become a great QA team?”

    The turnout was good with an average of 50m people from all kinds of companies in all kinds of roles related to QA. There were a lot of QA managers, but also producers, testers, developers, small developers to large publishers with a strong showing from Sony. (Their format QA is near San Francisco and QA folks from their operation in San Diego also had a strong presence.)