How To Avoid Pitfalls When Designing And Manufacturing An RF Automated Test Equipment System

It’s 7:00 p.m. on Friday evening. Your team is working furiously to finish the new test set in time for the test readiness review on Monday. This is a top priority for the director of manufacturing. It hasn’t been smooth sailing.

Thankfully, the software engineers worked their way through the latest “blue screen of death” to get the test program running in debug mode. The engineering-model brass board that was received three days ago is functioning. For the most part, everything is performing in accordance with the design specifications.

But, at the last minute, the customer changes his requirements!

This changes the product design. The revised design is now something that almost exceeds the laws of physics — and causes more than five specification changes. The team’s involvement started just three months ago. That’s when the program office finally approved the estimate and funding was provided. In that short time, the team was able to order the COTS equipment. Most of it was delivered and installed in the test system a week ago. A hardware engineer designed the unit under test (UUT)-to-test equipment interface test adapter (ITA). He did a great job, though the end result somewhat resembled a rat’s nest.

Rats NestDoing their best, the software engineers started development in the dark. They had to split their limited time between supporting existing test systems and working on this new development program. One of those existing systems failed two days ago and required the software team’s immediate attention.

The test system that failed had not been serviced for the past eight years, and one of the obsolete COTS instruments finally gave out. Plowing through the thousand lines of spaghetti code — written just before the senior software engineer retired — was a tough job. Somehow, the software team was able to find each equipment call, in each measurement, and replace it with the proper call for the new COTS equipment.

The old equipment is now patched up, but what happened to the new test system they were working on? You guessed it: the long delay from fixing the old system delayed the software team from finishing the code per the written schedule. This caused an immediate spike in overtime and stress amongst the entire team, and you are left to ponder why test engineering is continually brought into the design process far too late.

ATE Problems Occur On A Weekly Basis Around The Globe

Did this scenario sound familiar?

If you are a test engineer or test engineering manager, you have lived this. Automated test equipment (ATE) system development and manufacturing is a difficult job for many design teams. There are many pitfalls that cause stress and delays. However, most of these can be avoided and resolved if they are identified and brought to the surface long before a program begins.

This article examines the exact items that make life happy — or miserable — for test engineers. Which life each test engineer leads depends on the approach taken while designing a new system: the smart way, or the typical, hasty way. Further, my hope is to provide the impetus to facilitate discussions with upper management. The goal is to resolve the pitfalls first, make life better, and enable better company productivity.

How It All Begins — The Bid And Proposal Process

Most new system test programs begin with a customer proposal, and the average company provides a detailed proposal that includes a design concept, a cost estimate, and a projected schedule.

Frequently, the proposal uses “boilerplate” estimates for the cost of production test equipment, which usually are far too low. If your company allows you to help write the test portion of the estimate, feel fortunate. However, beware if your test services are bid too low. The prospective customer will come back for a “Best and Final Offer” and your estimate will be targeted for cost reduction. Then, you’ll be left with far too little budget or development time to complete the expected tasks.

Thus, test engineers need to lay out the exact reasons why system test is important. The challenge is to get in front of the program manager and talk test! Once that is accomplished, explain how a properly designed and funded test system will:

• Reduce test time
• Reduce manufacturing down time
• Reduce measurement errors, and
• Most important, provide repeatable test data that will assure customer confidence.

There is immense value in taking this step; the exact procedures must be identified and communicated as early as possible in the bidding process. Start now in educating your manufacturing director and other managers. When they hear of a new program from now on, hopefully they will push for the test department’s involvement from the very beginning of a new proposal.

Provide An Accurate Estimate

Don’t neglect little details in the budget! When producing a budget estimate, the common tendency is to focus on the large-cost items and to neglect the little details. The estimate must always include the cost of COTS equipment, the equipment enclosure, the control computer, and the ITA. Generally included is hardware and software design time.

Yet, sometimes, other items are forgotten or neglected. The other items tend to increase cost and lengthen the time it takes to complete a system. Ask yourself:

Did I include the things a good system needs?

• Have I included a good cable management system?
• Did I include a power distribution unit with the required safety items?
• Have I allocated enough time for a qualified technician to wire the ITA and ATE?
• Did I include time and funding to prevent the things that often plague system performance, such as:
  • signal crosstalk,
  • differential signals,
  • ground loops,
  • EMI/RFI problems,
  • cable loss, especially for high frequency RF/microwave signals?
• Did I budget enough time for software development?
• Is there adequate system debug and validation time included in the estimate?

Remember to account for money and schedule according to Murphy’s Law: “Whatever can go wrong, will go wrong."

Root Your Schedule Planning And Management In Reality

Do you have a reality-based, integrated master design and production schedule? This is essential to forecasting staffing requirements and allocations. A production schedule must integrate all programs into one comprehensive schedule. It must be accurate, tracked, and revised. And, when schedule slips occur, it must allow time to provide alternate work-around plans. Finally, the schedule must include time to support existing production test systems.

Once you have a reality-based, integrated master schedule, carefully examine it. Then, post it where everyone can see the progress and remaining work. Looking over the next three to six months, is there any point where staffing requirements approach or exceed 90 percent? What action will be taken when that happens? What are the contingency plans? Be sure to have a back-up plan.

Don’t discover the problem when it is too late! If the test department is shorthanded, it will certainly be at the most critical time: the last minute before the system is due!

System Design Is More Than “Rack And Stack”

To overcome price or schedule pressures, don’t yield to the temptation to cobble a few instruments together with some cables. Invest adequate time for proper planning and system design. Be prepared to calculate and communicate the actual cost of providing a well-designed test set, versus one that is been thrown together in haste.

Think about which measurement techniques are needed, and which available COTS equipment will best meet the need. Look at available production test floor space, and determine how much space the ATE system will take, and what it will look like. Be sure to determine the following:

Will the system be built with individual box-type instruments, or modular instruments?

• What form factor will you use?
• Which instruments and bus speed performance do you require for the measurements?
• Will the equipment be installed in an equipment enclosure, or will it be placed on a test bench?
• How can taking advantage of multi-functional instruments and switching minimize the ITA design?
• Look into the future. Think about what re-use this new ATE system may have for future requirements.
• Can changing the ITA allow for testing multiple products? ‘

Once the system is designed, calculate test times, measurement accuracy, and repeatability. Be ready to sell the system design to the program office and upper management. Be your own advocate.

Select The Right COTS Equipment

Measurement quality matters, making it vital to do your homework when selecting COTS gear to be used in an ATE system. Make certain your knowledge about available COTS equipment is current and unbiased. For example:

• Be sure to investigate multiple COTS suppliers.
• Examine instrument performance versus price from multiple vendors.
• Find out how long the target instrument has been on the market (to avoid your ATE system becoming prematurely obsolete).
• Find out the expected delivery date.
• Get a demonstration from the supplier. Be sure to evaluate whether the system can make the required measurements.
• Do homework on the COTS supplier; meet with their representatives and sales engineers, too.
  • How often have you seen the sales person in the past few months?
  • Do they keep you abreast of the latest offerings?
  • Are they knowledgeable about what they are selling?
  • How is the suppliers’ technical support?
  • Do they offer application notes?
  • Do they offer demonstration units to try out? You could even start your software development with the trial instruments.
• Does the COTS instrument fit into your design?
• Does it have extra capabilities that may be utilized if the test specification or measurement requirements change?

Embrace ITAs

An ITA is more than routine signal routing. High-density signal interfaces provide the opportunity to design and manufacture a very capable test system. Is the UUT composed of multiple sub-assemblies? Can one ATE system be used for testing the sub-assembly with multiple ITAs? Have you considered supplying a system self-test ITA?

Many questions about test failures and ATE troubleshooting can resolved with a self-test ITA and some software. A self-test ITA can resolve the age-old question about whether the UUT or ATE failed. Ask yourself:

• How can you calibrate/normalize your ATE system stimulus signals and measurements?
• How can you embed switching, signal conditioning, and auxiliary test points into your ITA?
• Would a firmware device, such as a field-programmable gate array (FPGA), simplify the ATE design or ITA?

Adding these items can reduce the amount of COTS required, and provides a quick way of UUT troubleshooting with external instruments.

Fixture Your UUT Carefully

Superior test fixtures result in better measurements and shorter cycle times. Just as companies tend to minimize the contribution of test engineering to the final product, test engineers often minimize the importance of a high-quality test fixture. A properly designed and manufactured test fixture secures the UUT to the test bench quickly and easily. This protects the UUT from being knocked on the floor or moved during measurements, both of which could result in measurement errors and potential damage to the UUT or ITA interfaces.

A high-quality test fixture can provide a method of quick connect/disconnect of UUT supply voltage, test signal stimulus, and measurement access points. It may include blind mate connectors, pogo pin interfaces, rapid clamping devices, and possibly even a thermal cycling platform for UUTs that require thermal test. Test fixturing is best left to a good manufacturing mechanical engineer. If a mechanical engineer is not on your team, you’re missing a key individual. When designing an ATE system, do not neglect this important part of the total plan.

Software Design Will Make Or Break Your System

The ability of software to make fast, reliable, and accurate measurements is critical. Well-designed software provides the user with an ATE system that is scalable, obsolescence resistant, and easy to maintain. It is critically important to study and understand the measurement science behind the design to select and maximize the capabilities of available COTS test equipment.

Software design should be well-structured, including provisions for measurement extensibility and hardware abstraction. It must be efficiently designed and include measurement modules, driver libraries, and a well-crafted, flexible, and visually appealing test executive. The bottom line is, it takes time and thought.

Good software requires more than a rudimentary knowledge of coding. A qualified and certified software engineer should design it. He or she will come (preferably) with extensive practical knowledge of test and measurement. If your team doesn't have a person with these capabilities, you will always have software that is inefficient and less supportable.

Know When To Ask For Help

Have a pressure-relief valve ready to prevent things from blowing up. As you look at the plan, analyze the integrated ATE production schedule, and review the team’s capabilities, be sure to look at choke points that may require resources that exceed your team’s size or expertise.

The best time to seek assistance is long before it is needed. Talk to the manager, program office, and team about acquiring help. Have you ever considered partnering with a company who specializes in test and measurement? A good outsource resource can add the capacity and capability to your team. A capable, experienced test systems company (sometimes called an integrator) can help fill in the voids.

All test system companies are not created equal. Their capabilities must go beyond just “rack and stack.” A well-qualified test systems company has an expert team that specializes in a broad spectrum of measurement system hardware and software design, and manufacturing. Many test system companies provide services ranging from a complete turn-key test system — designed and built from a customer-supplied UUT specification — to ITAs, cables, software, fixture designs, build to print services and all other ancillary test system components.

A good test systems company can partner with the internal team to help with software and hardware consulting. These services should be provided at a fixed cost and schedule. Thus, using an outsourcing strategy frees up the internal team to focus on work in progress, manufacturing test support, and other systems development.

Further, outside test system companies eliminate the hassles of daily management, benefits, and coaching, as well as overhead or continued employment, all while bringing day-to-day expertise to your system requirements. This unique expertise is not easily obtained by hiring new employees or expending your team’s energy trying to stay current with the latest COTS equipment and measurement techniques.

Even if you have never used or have no current plans to use a test systems company, it is important to know a good one. That way, when the need arises (and it will), you’ll have a resource to approach for a proposal.

To find a test systems company, search online for COTS equipment vendors. Most have partnering programs that recognize test companies with a proven track record in creating well-designed ATE systems. Go to trade shows and talk with test system company exhibitors. Study the company’s background, number of years in business, capabilities, size, portfolio of customers, and diversity of experience — especially in RF systems. Your effort will pay off in much greater productivity.