"Ansatz" is a fancy way to say that scientists make stuff up.

The most common formulation of physics is based on what are called differential equations, which are formulas that relate the rate at which things change. Some of these are easy to solve, some are hard to solve, and some can't be solved. It turns out that there's a deep reason why there's no universal way to find these solutions, because if that existed it would let you answer questions that we know to be uncomputable (thanks to Alan Turing).

But differential equations do have a very handy property: their solutions are unique. That means that if you find a solution, it's the solution. You can guess a solution, try it out, and fiddle with it to see if you can make it work. If it does, your guess is justified after the fact. That's what an ansatz is, a guess that you test. It's a German word that could be translated as initial placement, starting point, approach, or attempt.

Hans Bethe famously did this in 1931 with an ansatz for the behavior of a chain of interacting particles. His solution has since been used to study systems ranging from electrons in a superconducting wire that can carry current without resistance, to trapped atoms in a quantum computer.

There's a similar concept in probability, called a prior. This is a guess that you make before you have any evidence. Once you do make observations, the prior gets updated to become what's called a posterior. It's initially equally plausible for the universe to be explained by a Flying Spaghetti Monster or the Feynman Lectures on Physics; the latter becomes more probable once its predictions are tested.

Finding an ansatz or a prior is a creative rather rigorous process—they can come from hunches, or whims, or rumors. The rigor then comes in how you evaluate them. You could call this a hypothesis, but the way that term is taught misses both how these can start without justification, and how you initially expect them to be wrong but then patch them up.

My favorite approach to research management is "ready fire aim." You have to get ready by doing your homework in an area, then do something without thinking too much about it, then think carefully about what you just did. The problem with the more familiar "ready aim fire" is that if you aim first you can't hit anything unexpected. There's a sense in which everything I've ever done in the lab has failed at what I set out to do, but as a result something better has consistently happened.

Research progress is commonly expected to meet milestones. But a milestone is a marker that measures distance along a highway. To find something that's not already on the map, you need to leave the road and wander about in the woods beside it. The technical term for that is a biased random walk, which is how bacteria search for gradients in chemical concentrations. The historical lesson is just how reliable that random process of discovery is.

The essential misunderstanding between scientists and non-scientists is the perception that scientific knowledge emerges in a stately procession of received knowledge. As a result, ambiguity isn't tolerated, and changing conclusions are seen as a sign of weakness. Conversely, scientists shouldn't defend their beliefs as privileged; what matters is not where they come from, but how they're tested.

Science only appears to be goal-directed after the fact; while it's unfolding it's more like a chaotic dance of improvisation than a victory march. Fire away with your guesses, then be sure to aim.