Independent Assessment Test Results

In September and October of 2000, the first generation of the model and data used in InSite was thoroughly reviewed and assessed by a team of engineers and analysts from across the aerospace industry. Independent representatives from the U.S. Air Force, NASA, universities, the FAA, and industry spent five weeks entering their own system concepts into the tool and analyzing the tool's resulting estimates, including comparisons to actual data for existing vehicles. Twelve of the 17 independent assessors, representing a diverse mix of industry, government, and academic organizations, provided detailed feedback on their assessment of the modeling tool. All assessors reported that the software is a very useful tool for operations analysis and costing, including applicability to vehicle and spaceport design as well as education. Results of the assessment are summarized in the report below.

Based on the feedback from this assessment, the Vision Spaceport research partnership developed a second generation model. InSite is based on this second generation software.

The Independent Assessment Team kickoff briefing. Vision Spaceport Industry co-chair Kevin Ingoldsby (Lockheed Martin) conducts the briefing.

The Assessors

The group of experts participating in this assessment is notable for its diversity. The group was well balanced among private industry, government, and academic sectors, as well as among the applications they typically work with. These applications included a balance between military and civil, aircraft and space, satellite and launch vehicle, and between engineering, operations, and business.

Most of the responding assessors had considerable experience with modeling tools. Collectively, they listed 22 different models with which they have worked. The assessors averaged about 13 years of experience with modeling.

Overall Impressions

A few summary quotes from the "overall impressions" remarks:

§ … an excellent tool for evaluating operability/operations costs for space transportation systems § … the tool was easy to use, logically structured, and provides good importing/exporting features § Very useful tool, immediately helpful for our Space Operating Vehicle studies § Great concept…continue § The model does a good job of highlighting the things that drive operability and maintainability. § The model is an excellent training and educational tool… KSC should spend a concerted effort on using this tool as part of the 2nd generation RLV program

There was a general consensus that the model provides credible results. Credibility issues raised were the perceived KSC bias, statistical validity/significance and precision of the results, number of inputs, independence of element scores, and insufficient level of detail of operations input.

Detailed Assessment Feedback

Model Input

General consensus expressed that the model asked the right questions at the right level of detail. Several user interface improvements suggested.

Model Output

The responses in this area were almost completely focused on suggestions for improving the model's output forms and reports. There was general consensus that the color-coded figure-of-merit report was most useful, but all agreed that the output reports need more work.

Software Usability

The assessors generally praised the figure-of-merit approach and offered many suggestions for improving the software usability. There was a repeated request to show side-by-side comparisons of multiple concepts [which has since been added as a result of this feedback]. Not a single problem was reported for software installation and setup.

Commercialization

Virtually unanimous agreement that there is a potential commercial market for the tool in vehicle and spaceport design applications. The potential use as a educational tool was highlighted repeatedly, as well as the opportunity to use the tool as a standard for government evaluation of proposed future designs.

Notable Feedback Results

I think the model is an excellent tool for evaluating operability/operations costs for space transportation systems. The amount of input data required is fairly large, but I believe it is necessary, and I found that within a short time I became proficient at data input. Data import at the system or element level is an outstanding feature. I also liked the color-coded, order-of-magnitude output screen formats. I think your product fills a long-standing need in the space transportation industry.

Very useful tool, immediately helpful for our Space Operating Vehicle studies.

The model does a good job of highlighting the things that drive operability and maintainability. Major kudos for attacking the most difficult part of operability analysis and operations costing: given the absence of data, how does vehicle and ground system characteristics affect operability, supportability, and maintainability? You are a long way from having a usable cost model. May be closer to having an operations assessment tool.

I found the model predictions to be quite credible when I examined existing systems that I have some knowledge of, including Titan and Atlas. I found that the model accurately assessed the operability improvements that have been implemented by Titan and Atlas over the last 10 years.

Generally, FOM's tended to move in the expected direction and the expected order-of-magnitude for the Shuttle booster cases evaluated (FSB, ELRB, and RFS): a) Totals for RFS vs. ELRB element were higher for RFS. b) RFS's looked higher across the 7 scoring categories than the ELRB's (as expected due to additional systems, larger size, and infrastructure needs vs. expendable elements).

I think the model would be valuable for trade study evaluations. The Reusable Space Transportation Systems group at Lockheed Martin could benefit from the use of this tool over the next 5 years as we are designing new transportation system architectures. I will advocate the use of this tool to our systems engineering and integration groups.

I am interested in 8 hours from landing-to-launch turn-around time. Therefore, I would like to use this tool to help us decide what launch system has the greatest likelihood of helping us achieve that objective. This means the selection of fuel, stages, processing, etc from both the vehicle and facility perspective.

We need to develop a good baseline cost model of the space shuttle, taking into account all of the historical data. The 1971 era shuttle operations cost models predicted costs of $6 million per flight. That was off by two orders of magnitude. We need to do a study of where we went wrong with that modeling effort to avoid such a large error in the future. We need to be specific about what is meant by the term "operations." For example, does it include the sustaining engineering and logistics costs?

I agree with your position that operability/operations cost is primarily dependent on system/subsystem complexity, and the model input data provides a thorough characterization of all major ground and vehicle systems and subsystems.

I am not a designer but in general this model seems as good as one can presently have. The most important point is to be flexible to be able to incorporate future developments. This seems to be the case.

The model is an excellent training and educational tool for the multi-disciplines and system interactions involved with launch vehicles. KSC should spend a concerted effort on using this tool as part of the 2nd gen RLV development program. Attempts should be made to validate the model outputs. Produce reports for shuttle, Titan, EELVs, Delta, Atlas, etc and see if those users and developers agree with the results and how they compare to one another.

The capability to estimate startup and recurring operations cost would have serious market potential. ISE is interested in simulation based tools.

Yes, I would like to see some sort of arrangement to work with this effort to reach Air Force goals. This is most likely providing funds from the Air Force.

Obviously, a lot of good thought and analysis has gone into the model. I think that the Vision Spaceport team should work with a larger, NASA-wide effort to develop operations assessment and cost tools. I am certainly interested in future developments and would welcome the opportunity to work with the Vision Spaceport Team.

I think that this would be an EXCELLENT tool for graduate (and even undergraduate) education. The model and software would allow students to explore the consequences of technical design problems they learn about. They would be able to "test" designs in a "real" situation. The teachers would be able to give realistic problems to a level, which is presently hardly possible in this field. At the graduate level serious technological design could be explored. At the undergraduate level students would be enthusiastically motivated to work on "real" problems without having to know all the necessary technology. It would be worthwhile to seriously look into an "educational" version, which might emphasize presentation, simulation, etc.. Such a version should be offered in the educational market at a lower price. It would probably sell in many more copies and might overall be financially as worthwhile as the "professional" copy. This would also help to establish this as the standard.

#1 Vehicle design analysis -- The sensitivity of the vehicle design and subsystems to life cycle cost could aid designers and at the same time identify "that technology" that could dramatically reduce life-cycle cost. What a convincing way to put emphasis on a technology that could deliver a long-term lower cost system.

Areas to Improve

Many of these issues were addressed in version 2.0 of software that forms the basis for InSite. Others will be prioritized and developed through the User Group.

Input

Items missing: (1) There are no inputs for new technologies or processes or technology readiness levels of pending systems to see how the new items may affect the solution. (2) There are no inputs for assessing a "green field" spaceport approach. Model appears to assume that all work is built-up from KSC. 3) The "Power and Health Systems' tab does not appear to allow input for hydraulics and hypergolics vs. fly-by-wire.

Inputs focus on vehicle characteristics and functionality - What about operations and support characteristics such as engine removal rate? Some of the inputs were confusing. No definitions for the queries such as Propulsion and Power Integration. The Pop-up documentation varied between definitional and editorial. The definitions were good, I suggest that you capture the editorial stuff somewhere else. Missing inputs: Communications: L-Band, ability to communicate with both TDRSS and the ground; Navigation: radar altimeter, high resolution air data system, laser docking sensors; Thermal Protection: carbon-carbon TPS; Don't have an Element Purpose that neatly fits SSTO

Perhaps allowing the user to specify "baselined" vs. draft characteristics could be implemented and a summary that highlights that level of baseline could be provided.

The forms need to be consistent with standard windows format - you are making huge mistakes by deviating. e.g. placement of the cancel button on the left instead of the right.

Is there anyway to mark a tab so you can see which forms you have filled out.

Numeric Inputs should be automatically highlighted for replacement. Some inputs were confusing: seemed to be asking reliability questions in more than one place. Would be nice to have some way of tracking what has and has not been filled out. The large number of tabs make it easy to miss something. Some of the input groupings seem to be for convenience rather than logical (i.e. Mechanisms, Maintenance Requirements, & Cargo)

Output

As I mentioned at the workshop, reducing the built in multidimensional analysis to a one dimensional one in some of the final representations is somewhat misleading and not necessary. It gives up some of the power of the model.

Totals for concept by cost category graph (stacked column chart) --- Rather than $/Pound to orbit - $/Life-cycle (long haul)

I think all [output forms] are helpful, but should be presented in terms of sensitivity, ie, what is the cost (life-cycle) of selecting a hybrid propulsion reusable booster vs. a conventional liquid/solid throw-away booster.

A summary or comparison of the system complexity or technology level would useful as an addition.

Really would prefer side by side comparisons. Would also like to see a ranking of sensitivities to model inputs, which are drivers, etc..

The figures of merit have a lot of significant digits. Suggest that consideration be given to limiting it to 2-3 digits. Also consider using some descriptive words such as poor, average, good, excellent to go with the numbers. That might be helpful for high level decision makers.

Stacked column chart would be easier to read if it were larger and 2D instead of 3D. Side-by-side vehicle comparison would be a nice feature to add.

Need better print capability for FOM table. The printout on my old LaserJet III was almost eligible. The graph is unclear - hard to read. CVS file should have spaceport modules in same order as screen presentation. The summary tables for the FOM scores are meaningless (I know, they are just placeholders). Need a way of comparing different inputs for the same concept side by side.

Model

1)The value and use of the "Single Vehicle Performance" values is unclear. Is this data meaningful for something? How does one evaluate the one "total" system against another? 2)Flight and mission operations functions did not appear to be included (we were told these were included - but it appears that they are not. 3)The tool does not appear to be able to handle "Green Field" approach or a clean-sheet system comprised of newer technologies than represented in the model. The database is based on KSC data. How does that bias a new system? It was unclear as to if the model assumes evolution from the existing KSC infrastructure or everything is assumed to be newly acquired. This needs to be clarified. Can a "credit" or "penalty" be applied for use of existing assets? 4)It is unclear how "human rating is handled". How can a cargo-only vs. human rated system be objectively evaluated? Is the databased biased due to Shuttle and NASA history? 5) The "FOM Results for Vehicle Concept" summary sheet data did not change for any of the Shuttle booster cases evaluated (is there a software issue somewhere? In other words, every evaluation had exactly the same output data. Maybe some code was not linked or included?).

Other Models Used: What do you like best about the models used most often?

(Number of responses indicated in parenthesis)

1. User interface/ease of use (5) 2. Import/export capability (3) 3. Domain-particular specificity (2) 4. Ability to directly manipulate and review data (2)

What do you like least?

1. Data/algorithm limitations (5) 2. Lack of import/export/data sharing (4) 3. User interface (3)

Several noted that current ops models tend to oversimplify relationships between a vehicle and its operations; i.e., they are largely based on spacecraft weight, size and number of facilities.

Twelve independent engineers from industry, NASA, the Air Force, and academia reviewed the model software to assess its usefulness, validity, and areas for improvement. Their feedback - summarized above - was used to produce the second generation of the software that forms the basis for InSite.

Home Products & Services Technical Support News & Events Careers Contact Us