Mammal evolution: How ancient fossils are revealing the secrets of our earliest ancestors

A few years ago, I visited Liaoning Province in China, to study the region’s famous feather-covered dinosaur fossils. One day, after many hours of measuring bones and photographing feathers in a museum, I needed a break.

My Chinese colleagues exchanged a few hushed words of Mandarin and motioned for me to follow. “We have something secret to show you,” one of them said. “And it’s not a dinosaur.”

We left the museum, hopped into a car, and snaked through the narrow streets of Beipiao city, clogged with bicycles and noodle vendors. We pulled into an alleyway that opened into a small courtyard. Into a dark apartment we ventured, sidestepping a clutter of boxes and wooden crates.

One of my colleagues ducked into a side room and came out holding two rock slabs that fitted together like a jigsaw puzzle. On the surface was a brown smear, about the size of an apple.

Read more about prehistoric mammals:

• Skeleton of Australia’s marsupial lion rebuilt
• Oldest-ever liquid blood found in 42,000-year-old ice age horse

A surprise discovery

I looked closer. The brown stuff was hair, and there was a spinal column running down the middle. That’s when I understood: I was looking at a fossil mammal – a distant cousin of ours, which lived alongside the feathered dinosaurs in the Mesozoic Era, more than 125 million years ago. (The Mesozoic Era ran from about 252-66 million years ago and includes the Triassic, Jurassic and Cretaceous Periods.)

This particular fossil has yet to be described, and as far as I know, it is still gathering dust in that non-descript apartment in Beipiao. Maybe it’s a new species, or maybe it’s not, because over the past 25 years more than two dozen fossil mammal species have been found in the Jurassic and Cretaceous rocks of China.

Like the feathered dinosaurs, they were buried by volcanic ash and sludge, hence their abundance and pristine preservation. And, like the feathered dinosaurs, they are helping palaeontologists understand how a major group of modern animals – our own group – originated and evolved.

How evolution made the mammal

Mammals are unique among modern animals. Among other things, all mammals – from hamsters to humans – have hair, big brains, keen senses of smell and hearing, and differentiated teeth (incisors, canines, premolars, and molars).

We nourish our young with milk, are warm-blooded, grow fast, walk upright with our limbs underneath our bodies, chew our food with our complex molars, and are wizards at hearing high-frequency sounds via our chain of tiny middle ear bones and coiled cochlea.

These features, which together define the mammal ‘body plan’, did not evolve all at once. Instead, they developed piecemeal over tens of millions of years, beginning about 325 million years ago when the mammal lineage diverged from the reptiles. The mammal line – known as synapsids – came to dominate the Permian Period (299-252 million years ago), when all land was conjoined into the supercontinent Pangaea.

While some synapsids like the predatory gorgonopsians topped the food chain, another synapsid group, called the cynodonts, got smaller, moved into the shadows, and managed to endure the terrible extinction at the end of Permian that extinguished up to 95 per cent of all life. These cynodonts became the immediate ancestors of mammals.

New fossils, combined with cutting-edge analytical techniques, are revealing how mammals emerged from their cynodont ancestors during the ensuing Triassic Period. At first, these near-mammals were still quite reptilian in their biology and behaviour.

A stunning discovery of a fossilised mother Kayentatherium and more than three dozen offspring, described in 2018 by Eva Hoffman of the American Museum of Natural History, shows that these hairy, dog-sized creatures still had large clutch sizes and small brains.

Over time, however, they began to grow more rapidly. They developed bigger brains with a novel region called the neocortex for processing sensory information and rearranged their jaw-closing muscles for more powerful bites.

• Subscribe to the Science Focus Podcast on these services: Acast, iTunes, Stitcher, RSS, Overcast

Meanwhile, these proto-mammals were getting even smaller. This had profound effects on their biology, as shown by two recent studies led by Dr Bhart-Anjan Bhullar of Yale University and Dr Stephan Lautenschlager of the University of Birmingham.

They used computer animations and lab comparisons to study the differences between proto-mammals and modern mammals. As mammal antecedents were shrinking in size, the dentary bone of the lower jaw became the dominant jaw muscle attachment site.

It seems like a minor tweak, but it had major ramifications, freeing other jaw bones to morph into middle ear bones that could better hear high-frequency sounds, and permitting a new type of rotational jaw movement for more efficient chewing. It is at this point that we can say true mammals emerged.

• In pictures: ancient mammals that lived in the age of dinosaurs

How mammals diversified and thrived

Later in the Mesozoic Era, after the extinction at the end of the Triassic Period that ushered in the Jurassic Period, mammals really came into their own. Until fairly recently, it was said that mammals of this age were all tiny, boring, unsophisticated D-list actors in a dinosaur blockbuster. We now know, thanks mostly to the wealth of new fossils from China, that this is false.

Mesozoic mammals were small, that much is true. No known species was bigger than a badger, probably because dinosaurs already dominated the larger-sized niches. But the Jurassic and Cretaceous mammals became masters at surviving underfoot, diversifying into a remarkable array of species and ecologies.

There were tree-climbers like Agilodocodon; diggers like Docofossor that resembled moles; web-footed and beaver-tailed swimmers like Castorocauda; and vermin like Volaticotherium and Vilevolodon that glided between trees on their wings of skin. All of these mammals have been discovered during the last 15 years.

Read more about ancient mammals:

• What killed the Ice Age giants?
• Could we clone a mammoth or a dinosaur?

These different mammal species ate a wide variety of foods. Some, like the wolverine-sized Repenomamus, used their sharp teeth to devour baby dinosaurs, as recorded by a fossilised last meal in the stomach of one skeleton. Many others, however, started to specialise in two other food types: insects and plants.

They were simply taking advantage of what nature had to offer. During the Cretaceous, flowering plants entered the scene and diversified alongside their insect pollinators in an evolutionary waltz. To feast on the insect bounty, some mammals evolved a new type of molar tooth, with a series of cusps that formed a mortar and pestle for crushing and grinding.

• Subscribe to the Science Focus Podcast on these services: Acast, iTunes, Stitcher, RSS, Overcast

This special tooth – a tribosphenic molar – is a signature of two mammal groups that still survive today: the marsupials (whose minuscule babies develop in a pouch) and placentals (our group, which gives live birth to well-developed young). Both groups, along with the third surviving mammal lineage (the egg-laying monotremes), got their start in the Mesozoic.

But they weren’t exactly dominant. They shared ecosystems with a panoply of other families that flourished and then went extinct. Many of these, like the buck-toothed multituberculates, were extraordinarily successful.

Some multituberculates developed saw-blade premolars and intricate molars to cut and grind plants, whereas others like the Romanian species Litovoi, found by palaeontologist Mátyás Vremir and described in 2018, shrunk its brain but evolved super-keen olfaction to adapt to its latest Cretaceous island habitat.

The end of an era (literally)

If you were there 66 million years ago, on the final day of the Cretaceous, you probably would have heard various mammals scurrying through the underbrush… if you weren’t too focused on trying to avoid Tyrannosaurus rex, that is.

Some of those mammals might have been primitive placentals, but these were still rare, outnumbered by multituberculates and early marsupials. Then, in a moment, everything changed. A 10km-wide asteroid, speeding faster than a jet airliner, smashed into the Earth with the force of over a billion nuclear bombs.

Instantaneous tsunamis, wildfires and earthquakes gave way to nuclear winter and longer-term global warming. Dinosaurs like T. rex and Triceratops couldn’t cope, and all bar a few feathered-and-winged fliers went extinct.

• Subscribe to the Science Focus Podcast on these services: Acast, iTunes, Stitcher, RSS, Overcast

Mammals were one of the great survivors of this mass extinction. The story, however, isn’t as simple as often told. Many mammals died too, including several of the Mesozoic groups with non-tribosphenic teeth, and most marsupials, which came perilously close to annihilation.

Years of fieldwork in the western United States, led by Dr William Clemens of the University of California and Dr Gregory Wilson of the University of Washington, has clarified what happened: larger mammals and those with more specialist diets perished alongside the dinosaurs, whereas smaller mammals with flexible, omnivorous diets had a better chance of surviving. Among these were the placentals.

Life finds a way

Within a few hundred thousand years of the asteroid’s fury, the Paleogene Period was in full swing. Ecosystems had recovered and placental mammals were booming, with some species now nearly as large as cows.

I’ve spent several years collecting their fossils in the San Juan Basin badlands of New Mexico, with my colleague Dr Thomas Williamson of the New Mexico Museum of Natural History and our former PhD student Sarah Shelley, now at the Carnegie Museum of Natural History.

These mammals seem a little strange: they are stocky, their skeletons look quite archaic, and scans published earlier this year by our colleague Dr Ornella Bertrand of the University of Edinburgh show that they had smaller brains than today’s placentals.

In effect, these New Mexico mammals – species like Chriacus and Periptychus – were placental pioneers. They were the groups that survived the extinction, got bigger and more diverse in a dinosaur-free world, and set the stage for today’s 6,000+ species.

Among them we find the precursors of horses and deer, and of primates – our very own ancestors – beginning to carve out a life swinging from the trees, before eventually coming down, rearing up on our hindlimbs, growing bigger brains, and marching towards civilisation.

• This article first appeared in issue 349 of BBC Science Focus