Lesson: How to Design an Experiment

posted in: Lesson | 0

Take this lesson to learn how to design an experiment using the scientific method.

Meaning of Experiment Design

Suppose I am developing a math model of a falling rock climber (see Fig 1). The goal is to find the load on the anchor. I need to know the spring constant for a climbing rope, so I design an experiment for measuring k (spring constant), then I execute the experiment and process my data to give me a value of k.

Fig 1. When a lead climber falls, this can create a large dynamic load on the anchor (protection). Engineers size the hardware to handle these loads.

Next, I build a math model to predict the force on the anchor as a function of the relevant independent variables (falling height of the climber, the weight of the climber, the spring constant of the rope etc.)

To validate the math model, I design another experiment to measure the dynamic load of a falling object connected by an elastic cord to an anchor. Once this experiment is done, I compare the data with the math model. Nearly always, I need to loop back and do more experiments or revise my math model. That is, the process is strongly iterative.

Experiment design (ED) is the process for systematically planning, executing, analyzing and reporting an experiment. The ED process is nearly always iterative. The essence of ED is to let the data reveal what is most likely true (this is the scientific method).

An experiment is a process for making observations about the real world and then drawing conclusions are these observations. These observations are called data. The purpose of an experiment is to use information gathered from the real world to tell you what is most likely true.

Benefits of Using The Experiment Design Process

  • Super fun! (design, build, test, loop back ===> ton of hands on work) Super fast also.
  • This is the best possible way to learn engineering; the data from the real world gives you feedback on whether or not you are understanding; when you understand, your math model and the data align.
  • The data will reveal what is most true (math models often have mistakes or faulty assumptions)
  • Feel more confident in your data
  • Be able to explain and share your work with others openly (here is what I did; how does this look?)
  • Essential for credibility (technical work must be presented using the scientific method)

How to Use the Experiment Design Process

Step 1. Why. Figure out the reason for doing the experiment.

  1. Validate a math model?
  2. Measure performance to see if specs are met?
  3. Gather experimental data to build a math model?
  4. Characterize performance? [e.g. how well does spud gun work]
  5. Test design options? [which option is better: a? b? or c?]

Step 2. KC. Find and apply existing theory and knowledge. Take advantage of the collective wisdom of human kind.

Step 3. Make predictions (state hypotheses). This step is not always done.

Step 4. Figure out your experiment before you do it. Document your work.

  1. Goals. What data do you want to measure? How many data points?
  2. Ideas Generation
    1. how will you measure your data?
    2. what can bias your data?
    3. how will you calibrate your transducers
    4. etc.
  3. Analyze and select the best ideas
  4. Make a step by step plan for executing your experiment.
  5. Assemble all the materials, instruments etc. you need to do your experiment.

Step 5. Execute your experiment. Process the data.  Document as you go. Review your process and results. Decide next steps. Compare findings to your initial theory/knowledge. Go back read the literature some more.

Step 6. Loop back and run through the process again. The ED process is nearly always done iteratively. Much faster and better to do the process 2 or 3 times than to try to do it right in one step (this rarely or never works).

Knowledge Construction. After the iterations are done, see if your finding are consistent or inconsistent with your foundational theory. When there is an inconsistency, then bring this out. [this is how science is used to continually build and revise theory].

Reporting. After the iterations have been completed, report your findings in writing. Organize your writing with the scientific method (problem, goal, literature review, methods for doing my experiment, data, interpretations, conclusions). Report what the data says openly and honestly.

Top Tips For “Kick-Ass” Experimental Work

  1. Figure out your experiment before you do it. There is a super-simple, low-cost, fast way to run your experiment. Find this way so you don’t waste a gob of time and money.
  2. In any experiment, there are 100s of things that can cause bad data. Think profoundly as you design your experiment so that you can mitigate your risks.
  3. Do not over-plan your experiment. You can never anticipate all the problems that you will face. Instead, plan your experiment, get some data, process ==> repeat. Use the iteration process because the first time you run your experiment rarely works.
  4. Understand how your scientific instruments work. That is, what are the physics?
  5. Always expect your data to be wrong (it often it). Then, employ strategies to check your data.
    1. Calibrate your transducer.
    2. Run an experiment where you know what the data should be and see if you can reproduce the data.
    3. Triangulate. This means to measure the same thing in three different ways.
  6. Report your data with a spirit of humility and openess. Here is my data, I am not 100% sure if this is correct but here are my methods. Here are the checks I made
    1.  I calibrated my transduce like this …
    2. I triangulated like this …
    3. I validated my experiment like this ….
  7. Put your focus always on letting the data reveal the truth to you. Data do no lie. Let the data be your teacher and your guide.
  8. Be open to the possibility that “I made a mistake.” Thus, check your results in every way possible so that you can rule out everything that you can think of. Be your own worst critic. When I publish my experiment work, I know that my checks have been 10 times more rigorous than any one who ever looks at my work.

What is the Scientific Method ?

The scientific method is a belief system about the world. People tend to believe in science or to believe in alchemy.

  • Scientific Belief System. I believe in using data, evidence and reasoning to find out what is most true. The data will reveal what is most likely true. People with this belief system are skeptical of their own beliefs and results because they have had many experiences in which the data has altered their world view.
  • Alchemy Belief System. I trust my own beliefs and intuitions. I will say that I believe in science. I will use data and evidence when it supports my beliefs.However, people with an alchemy-based belief system are not open to data that misaligns with their world view. They will become angry, argue against the data, and reject the data. This is called defensive reasoning and it explain why people keep doing things in the same way. It explain why people have difficulty with learning, growth and change. It explains why organizations stay stuck (keep doing things the same way). 

In higher education, only a tiny percentage of the faculty and students appear to have adopted the scientific belief system. Similarly in industry, only a tiny percentage of professionals appear to believe in science. The reason for this is that the brain naturally causes the individual to feel like:

  • I am right
  • My ideas are really good
  • My beliefs are correct
  • I understand this
  • I am good at this
  • etc.

When a person experiences a challenge to their beliefs, or when someone tells them that they are wrong, then the brain goes into fight/flight mode and releases fear chemicals (e.g. cortisol).  These chemicals shut down higher order thinking and cause the person to respond with anger and to reject the ideas. Defensive reasoning is hardwired into the human brain; it cannot be turned off because it is biological. It is the brains way of protecting the individual from threat.

However, once we recognize that believing in alchemy and defensive reasoning is a natural process (biological), then we can make a choice to believe in science. That is, we can choose to use  data and evidence instead of emotion/tradition/opinion as the basis for finding truth.

The benefits of believing in science are that it opens up many new avenues of thinking, one learns much better and it is much better than arguing (instead of arguing, lets get data and see what the data says).

When Can the Scientific Method by Used.

The scientific method can be used in many contexts. This section presents examples.

When I am learning XYZ, I can see if I can explain the topic from memory to a peer. When I can do this, then I have some data/evidence that I am getting my ideas down.

When my team has three prototype ideas for a design, we can prototype the three ideas to see which one works better (this is much faster and easier than arguing).

When I build my math model, I can get data to see if the math model is matching what goes on in the real world. This allows me to build a super-simple math model because the data will reveal if the model is good enough or not good enough. If it is not good enough, I can add refinements and build iteration #2 of my math model. This idea is the method of all good modelers that I know.

When I want to learn something new, I can start by reviewing the data/evidence that is already published (i.e. start by doing KC). Then I can try out my new knowledge and get data to see how well it is working.

When I want to see if customers for my design will want to buy my design, I can get data that will reveal if I am on track or off track. This is the central approach for creating designs that succeed in the marketplace.

If I think, I am a “hot skier” then I can get onto a race course and see how my times compare with the racers. If I am truly a hot skier, then I will run the gates faster than anyone. Since my times are much slower than great racer I can openly accept that I still have lots of room to grow. Data allows me to open myself up for learning.

Summary.  The scientific method can and should be used in all contexts of professional life and personal life. It is simply makes sense to base ideas off of data and evidence (not off of tradition, emotion, and opinion).

Goals for the Learner

  • Explain what experiment design means.
  • List and explain the benefits of using experiment design.
  • Summarize the process for doing experiment design. That is, what are the steps? What do good practices look like?
  • Describe the two belief systems: science versus alchemy. Explain why believing in science is rare in people
  • Explain the benefits of believing in science.
  • Give real world examples of using the scientific method for authentic context such as:
    • learning
    • deciding if a product will succeed in the marketplace
    • testing a math model


Leave a Reply