Lesson 1 - What is Statistical Mechanics?

The Macroscopic World and Classical Thermodynamics

Look around at the chunks of matter all around you. By now you probably have a couple decades of empirical experience in the macroscopic world (macroscopic = ‘the world you can see’), and depending on how inquisitive you are, you may know a great deal about how matter behaves under a variety of conditions….

Matter comes in chunks with uniform properties. If you throw one of these chunks it will glide through the air mostly unslowed until it hits the ground, perhaps with a crash or shatter. These chunks of matter can be shaped and layered and bonded together to make complicated and useful composite objects like the computer I’m typing on. Some chunks of matter react violently and explode or turn into new materials when brought into contact with one another. A hot chunk of matter, like an oven coil, will make anything touching it hot too. Rubbing two chunks of matter together will make them hotter. Small changes to the temperature of a chunk of matter sometimes have dramatic effects, like a snowman melting as the temperature rises above the freezing point…

Some chunks of matter exhibit sophisticated responses to stimuli. Some process information. Some reproduce copies of themselves. Some make internal representations of themselves and their environment. Some read and write coursenotes about statistical mechanics. Your experience gives you a rich intuitive model of the phenomena exhibited by macroscopic chunks of matter.

The scientific study of the mathematical laws governing heat and macroscopic matter is called thermodynamics, and classical thermodynamics developed over the 18th and 19th century to give empirical mathematical descriptions of these phenomena. The quantitative patterns observed are summarized in the laws of thermodynamics. Roughly, these laws state:

• Zeroth law of thermodynamics: If two chunks of matter are brought into contact (usually by touching), they will exchange heat until after a sufficiently long time they reach a steady state called thermal equilibrium, with stable macroscopic properties. The equilibrium state does not depend on the details of how the chunks of matter touch, and two chunks of matter in thermal equilibrium are said to have the same temperature. Practically, temperature can be measured by bringing a chunk of matter into contact with a much smaller reference system, for example a mercury thermometer. When two chunks of matter have reached thermal equilibrium, they will have the same effect on the reference system– in this example, the height of the mercury column will read the same.
• First law of thermodynamics: Heat is a form of energy stored inside matter, and can be converted into macroscopic mechanical energy. Mechanical energy means the kinetic and potential energy associated with macroscopic motion and forces – for example, heat from combustion in a car engine is converted into the kinetic energy of the motion of your car down the road. Mechanical energy can also be converted into heat, for example when a car slows on a road due to braking, the macroscopic kinetic energy is converted into heating of the road and brake pads. The total amount of energy, heat + mechanical energy, is conserved in any process.
• Second law of thermodynamics: The second law of thermodynamics says roughly that energy and matter tend to spread out. If you pop a balloon, the molecules that were in the balloon tend to spread out through the entire room. If you heat one corner of your house with a fireplace, the heat tends to spread through the entire house (and unfortunately, eventually, to the outdoors as well). This idea of “spreading out” can be made more concrete and quantitative by defining a quantity called entropy, which we will define concretely later. For now, we’ll simply say that this tendency of energy and matter to “spread out” can be quantified into a mathematical statement that the entropy of any isolated system always increases.

The Zeroth Law of Thermodynamics and the Second Law of thermodynamics are connected by the idea of thermal equilibrium – when two chunks of matter come into thermal contact, that means energy that was previously trapped within one system, can now spread out between the two systems! The equilibrium state corresponds to the state of maximum “spreading out” of energy between the systems, or more accurately, the state of maximum entropy! Once we define entropy, we will be able to use this connection to define temperature in terms of entropy.

The Microscopic World and Statistical Mechanics

From your studies and general scientific cultural bombardment, you probably have also learned a thing or two about the microscopic world in the past couple decades. When you look closely, the world is not at all what it first seems. Stable, solid and continuous matter gives way to bouncing shaking atoms, often moving and colliding at 100s of meters per second. These atoms interact with each other very strongly at distances of around 1nm to form molecules or bonded structures, and solids and liquids typically contain atoms packed together at a density of roughly 1 atom per \(nm^3\) (or roughly \(10^{21}\) atoms per \(cm^3\)), while gases under atmospheric conditions are typically hundreds of times less dense.

When two molecules collide in a gas, they act a bit like infinitesimally tiny billiard balls, following newton’s laws of motions and the conservation of energy and momentum; when they interact in closer quarters, for example in chemical bonding or the formation of solid and liquid states of matter, they do an entirely different dance, and quantum mechanics and the nuanced interactions of electron clouds becomes important. In living matter, well before you reach the nano scale of atoms, a world of complex cellular processes emerge at the scales of hundreds of microns, containing within them complicated biomolecules and sophisticated molecular machines. In thinking matter, this cellular world joins into a nervous system that supports entire emergent electrochemical universes of feelings, personalities, and ideas.

Statistical Mechanics is the study of how the phenomena of macroscopic world, known both from our intuitive experiences and classical thermodynamics, emerge from the very different components and dynamics of the microscopic atomic world that lies beneath. There are a few central puzzles in this connection:

1. Although we know the microscopic laws governing atoms extremely well from studying systems of a few atoms, it is practically impossible to solve the coupled equations of motion for the \(\gtrsim 10^{20}\) atoms found in any macroscopic system, even using numerical methods and supercomputers. In addition, even if we could solve these equations, it would be very onerous to precisely measure or prepare the initial position and momentum of every single one of those atoms to make a prediction about their future behavior (and because of the chaotic nature of these systems, we would need to measure the initial positions and momentums to enormous accuracy to even make a prediction a few seconds in the future). How then in practice can we use our knowledge about the microscopic behavior of small numbers of atoms to predict and understand the behavior of macroscopic matter?
2. If matter in equilibrium is made of fast moving atoms bouncing around and shaking, separated by empty space, why do we perceive matter to be continuous and static in almost all of our experience?
3. What is the microscopic nature of heat and temperature? For some time it was thought that heat might be some type of fluid that flows within matter, yet in the microscopic theory of atoms no such fluid exists! It turns out that heat and temperature are not concepts that make sense for a single atom, and emerge only in describing the average behavior of systems of many atoms! It was eventually understood that exchanges of heat can often be roughly understood as changes in the kinetic and vibratory energy stored in the motion of atoms within a chunk of matter– we will start from here and eventually give a more precise mathematical description of heat in relation to entropy.
4. Even more mysterious than heat to turn of the 19th century physicist was a quantity called entropy. In classical thermodynamics, entropy was defined by a relationship between heat exchanges and temperature, and found empirically to obey the Second Law of Thermodynamics, namely to be a quantity that increased in any process. Qualitatively, it was observed that entropy also corresponded to a notion of energy and matter “spreading out” as we have all ready described. But what does energy and matter “spreading out” mean in terms of the atoms that compose the matter? And how does the mysterious second law emerge from the simple rules that govern the motion and exchange of energy between atoms?

Statistical mechanics solves these problems in a clever way. Instead of studing the motions of individual molecules at specific instants of time, in statistical mechanics we study averages over the motion of many molecules over long periods of time. This process of reducing the many microscopic degrees of freedom describing the motion of many microscopic molecules to justs a few macroscopic degrees of freedom describing the average motion of the molecules is called coarse-graining (#CoarseGraining).

These average quantities turn out, in many cases, to be exactly the macroscopic quantities that were empirically studied in classical thermodynamics – our human senses interact with the macroscopic world on timescales of milliseconds and distance scales of hundreds of microns, and this naturally leads to averaging over atomic scale effects. For example, although the pressure of a gas in a balloon feels like a constant push when you press on it, your sensation is actually a moving average implicitly performed by your nervous system over the trillions of collisions that occur between atoms in the gas and the surface of the balloon near your finger within a few milliseconds. The pressure that was considered a steady quantity in your macroscopic experience and in the mathematics of classical thermodynamics, becomes a fluctuating average when we consider it microscopically. We will find that when we look quickly, closely, and carefully, quantities like pressure that seem homogeneous in time and space are actually fluctuating rapidly. The idea of equilibrium becomes a statistical notion (#StatisticalEquilibrium) – in equilbrium, the macroscopic properties like pressure and temperature are not actually steady, but rather fluctuating continuously around steady mean values.

At the scale of your senses, these statistical fluctuations are generally too small to observe, and we can focus on simply describing the mean behavior. However, when we look at systems closer to the atomic scale, these fluctuations become important and we will have to describe the fluctuations quantitatively to understand their behavior. The phenomenon of Brownian motion is a classic example of this – sufficiently small pollen particles in a liquid can be observed under magnification to jitter due to the fluctuations in the pressure of the fluid around them:

Within the time resolution of the human eye, many collisions of fluid molecules with the pollen particles occur, so it is important to realize that this phenomenon is not a direct observation of the randomness of individual molecular collisions. Rather, we are observing the fluctuations averaged over many individual collisons. We will see that when we average over longer times and more molecules, these fluctutations become more and more suppressed (which is why you don’t observer brownian motion of your breakfast cereal floating in milk). Conversely, when we consider even smaller systems like single biomolecules in a cellular environment, and shorter timescales like those involved in chemical reactions, these fluctuations will become even more important.

Why study Statistical Mechanics?

Practical Questions:

Classical thermodynamics developed to address a number of practical concerns:

• How are the pressure, volume, temperature, and density of a gas related to each other?
• How do engines convert heat into useful work? For example, how effficiently can energy unlocked in burning coal or nuclear fuel be used to spin a steam turbine to generate electrical energy?
• How does the temperature, pressure, and concentration of reactants and products affect which chemical reactions will occur? For example, under what conditions will methane and oxygen spontaneously produce carbon dioxide and water (and some extra heat)?
• Under what conditions does matter transition between the solid, liquid, and gas phases? For example, how does the freezing point of water depend on the salinity and pressure?

These are very practical and useful things to understand in detail if you are a chemical or mechanical engineer, but honestly I find them a bit boring (a little over a decade ago I tried to become a mechanical engineer, but it turns out I love the matter of ideas more than I love ideas about matter :). Nonetheless, we will study them because some of you may be creative and practical people who will need these tools in your future endeavors, and because they also make excellent case studies where we can compare the macroscopic picture from classical thermodynamics to the microscopic picture developed in statistical mechanics.

Impractical Questions:

Perhaps not surprisingly, these very practical questions also lead to some deeply interesting ‘big’ questions about the world, and we will have a lot of fun with these:

• Can perpetual motion machines exist?
• Is the universe destined to end in a cold lonely heat death?
• Why does the future look so different from the past, when the microscopic laws of physics do not distinguish a direction to time?
• How does life tame the 2nd law of thermodynamics to locally decrease entropy and build complicated ordered structures?
• What are the fundamental limits on the efficiency of information processing systems?

Some of these are real braintwisters and were the source of quite a bit of consternation and philosophical debate for years. They are great topics for cocktail parties.

Wider Connections:

The problem of statistical mechanics is to find a way to compress the microscopic atomic information about a system into a few relevant variables– for example the positions and velocities of \(\sim 10^{20}\) molecules in a gas can be efficiently summed up by just four parameters, the pressure P, volume V, total number of molecules N, and temperature T, which obey a very predictive relationship with each other. This process of compressing the microscopic information is another way of describing coarse-graining, and rules describing the relationships between coarse-grained variables, like the ideal gas law, are called an effective theory.

However, it’s possible to widen statistical mechanics beyond the scope of atoms and matter. The underlying question of statistical mechanics is, given our knowledge of the small scale interactions or correlations between the building blocks of a system, how can we find the predictable large scale behaviors of agglomerations of large numbers of such building blocks? In the narrow application of statistical mechanics, the agents are atoms and the large scale behaviors are the classical thermodynamics of macroscopic objects, but the techniques and lessons learned with this application can be applicable to a wide variety of large complex systems– what can we learn from the dance of atoms in macroscopic matter that may help us understand the dance of humans in society? The collective dance of neurons in the brain and artificial neural networks? The dance of businesses in the global economy? While statistical modeling techniques have been long used to study these types of systems, the application of techniques and lessons from statistical mechanics to these kinds of systems is still a rather new field.

In the next section, we will describe in more detail how statistical mechanics can be viewed as a an effective theory for a coarse-grained description of the atomic world, and whenever possible we will frame our studies of statistical mechanics in terms of this wider point of view.

Personal Philosophical/Poetic Motivation

Finally, let me wax philosophical for a moment about why you should study statistical mechanics. Studying statistical mechanics is a practice that has changed how I see the universe and my self in it. When I practice seeing a macroscopic world secretly composed of atoms, I see the common ground from which we try to build our separate selves, and the awe-inspiring nature of the layers upon layers of complexity on which we sail as human beings. When I move my fingers to type and see that the mostly random motion of atoms can be corralled into purposes invisible at the atomic scale, I marvel at my own random human motions living in an incredibly complicated social and earth system. When I see through this lens and watch my niece delighting in falling snow, I can not help but share in her wonder at the world around us.

The Billiards Model

Many of you are probably familiar with the game of billiards, which is played on a flat table with colliding colored balls:

We will use this simple system to build some concrete understanding of the terms we have introduced in this lesson, and in class we will use simulations of this system to start studying coarse-graining, the second law, and equlibrium. While Billiards is only played with 22 balls, we will also study simulations with thousands of balls and eventually analytically study the properties of Billiards with \(\gtrsim 10^{20}\) balls as a model of real world gases made of \(\gtrsim 10^{20}\) molecules.

Usually we’ll study a simplified version of billiards with the following idealizations:

• The table is perfectly frictionless so that the balls can slide without rolling, and we will assume that the balls actually never rotate (so it’s a bit more like air hockey than billiards).
• The collisions of the balls with eachother and with the table are perfectly elastic (no energy is lost), and the momentum is transferred only in the normal direction of the collision (you have probably worked through this example before in a intro physic class, here is a quick refresher if you want one !! – we’ll use similar code to simulate this billiards model in python, but you won’t need to be able to write it yourself.)
• There are no holes for the balls fall into, just flat walls to bounce off of.
• We’ll mostly consider this model in two dimensions for simplicity, but occasionally we’ll generalize to three dimensions, where you can imagine the billiards bouncing around in a cube without any gravity.

You might call this “physicist’s billiards” –in reality, friction and inelastic collisions cause the balls on a typical billiard table to slow down and come to rest after about five seconds, but in physicist’s billiards, once started, the balls will bounce around forever (obviously there’s a few reasons you can never find a bar with a good “physicist’s billiards” table). We’ll also call this the “hard-shell” model sometimes.

Now, let’s map some of the ideas we have introduced to this system.

The microscopic degrees of freedom for this system are simply the variables that describe the motion of the balls. To describe each ball we need to specify its position on the table (\(x\) and \(y\)), and its velocity (\(v_x\) and \(v_y\)). So, if we have \(N\) balls on our table, we need \(4N\) variables to describe the microscopic state of the table. Note that we’re using “microscopic” in a more nuanced way here! While originally this term comes from the idea that we can’t see the atoms which are the fundamental building blocks of macroscopic matter, we can refer to the basic building blocks of any system as its “microscopic” degrees of freedom – and in this case we have a fundamental descrition of our idealized Billiards game just in terms of the positions and velocities of all the balls and their interactions, so they are our microscopic degrees of freedom.

The microscopic laws in this system are the rules for how the balls move and interact with eachother, which in this case are simply that the balls move in straight lines at constant velocity until they undergo elastic collisions with the walls or eachother.

What type of macroscopic properties of this system could we define? These variables need to be coarse-grainings of the microscopic variables. We might look at the average velocity of the balls of the system, or the number of balls on one half of the table, for example. It turns out that the average kinetic energy of some set of balls in the system will be an interesting coarse-grained variable that we will study in more detail to start to give a quantitative definition of temperature. In the Billiards case, the term macroscopic again isn’t quite literally accurate, since we can all ready see the individual billiard balls easily! However, we’ll soon imagine shrinking our billiard balls down to the size of atoms so that these averaged properties are the only easily observable quantities.

What about the second law and equilbrium? How do the balls spread out on the table? How does the kinetic energy spread out between the balls? Typically in Billiards, all of the energy starts in the cue ball before it collides with the other balls at rest. In our idealized version of physicist’s billiards, this energy does indeed tend to spread out into all of the other balls, and the balls do spread out over the table! We’ll study this with a simulation and then later use this example to quantitatvely define what is meant by “spreading out” in general. We’ll also see some of the important properties of this equilibrium:

• The microscopic details of the initial condition of the balls do not affect the equilibrium state of the macroscopic properties – you can imagine hitting the cue ball in many different directions or configuring the balls in many different initial positions, but the energy and positions of the balls will be distributed in the same way in equilibrium no matter what the initial conditions.
• The microscopic details of the collisions themselves do not affect the equilbrium state of the macroscopic properties! We have given a simple rule for how energy and momentum is exchanged in the collisions in the Billiards system, which we will use in our simulations, but the macroscopic properties would have the same equilibrium behavior for many different possible collision rules! For example, you could say the balls interact via a repulsive dipole force that repels them at a long distance from eachother instead of simple hard shell collisions. Or you could imagine instead of little balls, you have little tiny elephants that dance with eachother whenever they bump into one another. As long as energy is exchanged and conserved in the interactions, and the interactions happen sufficiently rarely, then we will find the same behavior of the macroscopic properties! This is an example of a phenomenon known as universality – there are many different possible microscopic systems that can have identical macroscopic behavior. This will actually be very good for us, because we are ultimately going to use our billiard model as a model for gases made out of real atoms, but the collisions of real atoms are much more difficult to simulate! Later on in the semester, we will develop some theoretical intuition for why this example of universality occurs.

Notes on topics not covered in this course

Basic dynamics of heat

The reader is assumed to be familiar with the basic mechanisms of heat transfer (conductance, radiation, and convection) from prior coursework. Because we are focusing on equilibrium behavior, we will not actually discuss the dynamics of heat transfer much, but it is useful background knowledge. The reader is also assumed to have prior familiarity with the concepts of heat capacity and specific heat capacities, and the Kelvin temperature scale.

Out-of-Equilibrium Statistical Mechanics

In this course, we are focusing on equilibrium Statistical Mechanics, which means we are studying the properties of systems that have had enough time to reach equilibrium. This includes something called quasi-equilibrium, when we imagine changing systems that have reached equilibrium so slowly that they approximately stay in equilibrium (for example, extremely slowly compressing a gas with a piston). In practice, the time scales for a system to reach equilibrium are quite fast and quasi-equilibrium is a good approximation for many processes in the world around us.

We will also discuss the idea of local equilibrium, where over short distance scales a system may be approximately in equilibrium, but over longer distance scales it is inhomogeneous. For example, sound waves traveling in a pipe are well described by local equilibrium – if you wait long enough, the sound wave will eventually dissipate and the system will reach a true equilibrium with a homogeneous density. However, even before this has happened, if you look at distances shorter than the wavelength of the sound and time scales shorter than the period of the wave, then you find the system behaves approximately as though in equilibrium.

While we will not go beyond these approximations in this course, we will discuss further some of the timescales involved in the approximation (i.e., how long does it take to reach equilibrium? how slow is slow enough for a quasi-equilibrium process?), and some of the ways they can break down.

Quantum Mechanics

There is a very interesting relationship between quantum mechanics and statistical mechanics! Statistical mechanics is a strange exception in physics, because in many ways it is actually easier to derive the rules of statistical mechanics directly from quantum mechanics than from its classical limit – essentially quantum mechanics involves discrete rather than continuous systems, and it’s simply easier to work with discrete probabilities than continuous probability distributions. Several textbooks present statistical mechanics starting from the microscopic point of view of quantum mechanics. However, due to the mixed background of our class, we will work from the classical physics microscopic point of view, studying classical systems like the ideal gas and the Ising model. While quantum effects are important in molecular bonding and chemical reactions, these can be abstracted away into a classical picture once we know the energies of different molecular states, and we will be content with this approach.

Historically, some of the first evidence for quantum mechanics actually came from studying the statistical mechanics of electromagnetic radiation (i.e, light), and quantum mechanics was also important for understanding the proper normalization of the entropy. We will briefly treat both of these topics, as they require very little background in quantum mechanics.

There is another fun connection beyond the scope of this class between statistical mechanics and quantum field theories through the topic of renormalization. Renormalization is a way of systematically and repeatedly coarse-graining systems that was developed for studying phase transitions in the Ising model, but became very important for interpreting and establishing the quantum field theories that describe high energy particle physics. Even further beyond the scope of the course, but interesting to mention, are some newer connections between quantum mechanics and statistical mechanics which appear in the study of black holes and quantum gravity. Blackholes appear to obey an effective theory of thermodynamics, although their microscopic components do not appear to be the particles we are familiar with. Moreover, entanglement in quantum information is typically measured using a generalization of the concept of entropy in classical statistical mechanics, and there is growing theoretical evidence that gravity and spacetime itself may be effective theories that correspond to coarse-grained descriptions of highly entangled quantum mechanical systems. If you’re one who is looking for weirdness in the small scale structure of the universe, there is no need to stop at atoms, and some of the tools you learn in this course will be useful for continuing your journey down the rabbit hole!

Condensed Matter Physics

Condensed matter physics is essentially the study of the properties of all matter that isn’t a gas, and includes studies of exotic phases like superconductors, detailed studies of phase transitions, models of electrical and thermal conductivity and optical properties, electron band structure, crystal lattice structure, etc. What we will learn in this class, especially when we study the Ising model, provides the foundation for advance study of condensed matter systems. However, we will not have time to delve much into some of the classic advanced topics like Landau theory, correlation functions and linear response, critical exponents, renormalization, etc. that might be found in a year-long course on statistical mechanics.