Humans are fantastic cheaters

When I started my PhD just over 20 years ago, my first brain-scanning experiment involved a simple short-term memory task: various red squares on the screen would blink blue. Then, after a few seconds, the volunteers would try to recall the novel sequence by pressing the squares on the screen in the correct order:

I made what I thought was an utterly trivial change from the previous experiment my PhD supervisor had run, turning the random-shaped grid of red squares into an orderly one. It turned out to be anything but trivial. First, the brain activity was very different. Furthermore, the volunteers approached the task in a radically altered way: Instead of a simple short-term-memory task, the subjects were taking advantage of the grid to find the regular shapes and patterns within the sequence. It was a powerful mnemonic trick that significantly improved their performance:

I swiftly learnt from my own experiments and the occasionally unexpected results of my research colleagues that such "cheating" was not an isolated incident. Far from it. Humans are pesky creatures where such experiments are concerned – they are always looking for tricks to make the task easier, which can, at times, ruin our carefully designed experiments. But such behaviour also reveals something very deep about human thinking and consciousness.

The briny taste of a sound aiding memory

Around this time I started to become interested in synaesthesia, which, I soon realised, might, in part, be capturing the same phenomenon. Synaesthesia is a relatively common condition (perhaps 1 in 20 of us has it) where we superimpose additional perceptual features onto sensory items. Perhaps you, reading this now, are a synaesthete: you might consistently link a colour with specific letters, numbers, days of the week, or months of the year? Or perhaps numbers have a particular spatial location for you.

One of the most fascinating synaesthetes in history is Solomon Shereshevsky, born in Russia towards the end of the 19th century, and vividly described by Luria in his book "The Mind of a Mnemonist". Shereshevsky was a relatively normal man, except for two key traits. The first was his exceptional memory. For instance, Luria reports one example where Shereshevsky listened to several stanzas from Dante’s Divine Comedy in the original Italian, a language he had no experience of. He was then given a surprise test for this 15 years later and recalled all words perfectly. How did he achieve such a prodigious memory? The answer lay in his strategic use of a combination of visually striking semantic images and, probably heavily linked to this, Shereshevsky’s second peculiarity: his extreme form of synaesthesia. To give a literal flavour of his synaesthesia, Luria reported what would happen if Shereshevsky were played pure tones of varying intensities:

"Presented with a tone pitched at 2,000 cycles per second and having an amplitude of 113 decibels, S. said: 'It looks something like fireworks tinged with a pink-red hue. The strip of color feels rough and unpleasant, and it has an ugly taste—rather like that of a briny pickle . . . You could hurt your hand on this.' "

It seemed likely to me that Shereshevsky was carrying out the kind of mental "cheating" that my own volunteers were performing, just on a far grander, more ubiquitous scale, both in technique and, correspondingly, in results. Occasionally in Luria’s book this was made explicit, as Shereshevsky would describe mnemonic tricks to aid his memory. These tricks would either rely on Shereshevsky’s amazing visual imagination, or on his rich synaesthesia. However, our main source of Shereshevsky’s characteristics are from this slim popular science book written half a century ago, rather than research papers, so there are many questions that remain, and, ideally, we’d need to study a modern equivalent.

A landscape of numbers

A few years later, in 2005, I was fortunate enough, briefly to have just such an opportunity with Daniel Tammet, on whom I carried out some behavioural experiments, and one brain-scanning study, in collaboration with Simon Baron-Cohen 　. As with Shereshevsky, Tammet appeared to have an exceptional mind. For instance, he had just set the Pi European record by reciting 22,514 digits from memory. He claimed to be able to learn a new language in a week (this was successfully tested in a TV documentary in Iceland). Furthermore, he claimed to be a mental calculator, for example dividing one double digit number by another, and giving the answer to 100 decimal places. In my own testing of Tammet, his digit-based short term memory scores on my tests were exceptional, and it was clear he had a very unusual approach to memorisation generally.

Daniel Tammet (Image credit: De Lorelei, Wikimedia Commons).

If Shereshevsky memorised the numbers 1 2 3 4, it would be trivial to him, but the obvious external pattern was something he couldn’t detect. I found a similar result with Tammet: if I gave him some regular sequence, such as 8 6 4 2 9 7 5 3 (descending even numbers, then descending odd numbers), he’d have no problems memorising it, but failed to notice the patterns that my other volunteers did. The reason in both cases, I believe, is that they are so focused on the internal synaesthetic patterns they experience, it is such a successful, habitual trait, that they fail to detect a structure that would be obvious to the rest of us. For, just as with Shereshevsky, Tammet was an extreme synaesthete, especially for numbers. And many of his mnemonic tricks relied on his number-based synaesthesia. For Tammet, each of the first 10,000 numbers have a unique synaesthetic feel, which includes colour, height, shape, and texture. Tammet described to me how mental arithmetic involves first having the numbers turn into his synaesthetic mountain landscape, then following some operation, he converts what he sees back into numbers to arrive at the answer. It’s this internal perceptual structure that seems to drive his numerical abilities, which is a powerful approach, even if on occasion it makes him oblivious to some obvious external pattern.

One advantage we had with Tammet over Shereshevsky is that we could take a full history and give him a clinical psychology assessment. Tammet, by his own admission, is not neurotypical. From childhood, he had autism, and although he is high-functioning and developed various tricks to minimise some of his autistic symptoms, it’s nevertheless clear that this aspect of his personality further contributes to his abilities and, possibly, his synaesthesia. Was Shereshevsky also autistic? Possibly mildly so, given that faces were one blind spot for him, but we have no real way of knowing.

The piano calendar

At the time, I was concerned that Tammet was only a single case study, and hence it was difficult to draw too many conclusions from him. But more recently another research group has studied a second person with surprisingly similar features. FC, as he is known in the literature 　, has various abilities, including perfect pitch, powerful calculation skills, and an exceptional memory for certain topics. He is a talented pianist, with superior memory for melodies. He can tell you what day of the week corresponds to a particular date, and is rapidly able to multiply four digit numbers together. At least some of these impressive skills are underpinned by his synaesthesia. For instance, he links the seven white keys within an octave on the piano with the days of the week, and the 12 black and white keys of a full piano octave with the months of the year. His mental arithmetic also follows a similarly circuitous route, as he converts all numbers into times (hours, minutes, seconds) to work on the calculation that way, and then back to normal numbers again to give the answer. With both Tammet and FC, it might seem that this conversion process is a wholly inefficient way of trying to remember or manipulate numbers. However, just as with my volunteers turning sequences of locations into regular shapes, what matters here is that we start with something abstract and somewhat arbitrary, and it helps enormously to impose some mental structure on these stimuli. That’s often the best route to memorisation and expertise.

Semantic Synaesthesia

Both Tammet and FC suggest that synaesthesia can be imbued with semantic information. But are these two case studies typical of the wider synaesthetic population? Historically, it was assumed by many that synaesthesia was entirely an automatic perceptual condition, and that the colours connected to letters, as an example, were fixed throughout life. However, a picture is emerging that the specific extra features synaesthetes experience are both more pliable than this and are heavily connected to meaning. Indeed, this semantic structure might be an important factor in people acquiring this particular flavour of experiencing the world in the first place.

One study that beautifully demonstrated how saturated with meaning synaesthetic experiences can be involved studying the most common type of synaesthetes, that of grapheme-colour synaesthesia, where numbers or letters are associated with specific colours. What Julia Simner and her group found was that if synaesthetes become depressed, then these colours take on less bright and less colourful shades, consistent with the depressed mood of these patients 　. It’s as if their entire conscious semantic world is influencing their synaesthesia.

The links between synaesthesia and autism

Given that common versions of synaesthesia are similar to these extreme case studies of Tammet and FC in their semantic content, just perhaps differing by degree, this raises a secondary question of whether synaesthesia usually also has links with autism, given that both the extreme synaesthetes Tammet and FC are autistic.

FC was studied more extensively than I was able to with Daniel Tammet, and it was found that not only is he very clearly on the autism spectrum, but there is a significant increase in prevelance for both autism and synaesthesia in his extended family.

Indeed, although this family study is the most in-depth in the literature to demonstrate an autism-synaesthesia link, other larger-scale studies have shown a clearly increased incidence of synaesthesia in the autistic population 　.

It is not just that these two conditions co-occur, but that they also share various key features, where the senses and thinking style of those with synaesthesia can resemble similar traits in those on the autistic spectrum 　. For instance, both autistics and synaesthetes can have hypersensitivity to sounds, sights and touch (intriguingly, many autistics see flashes of light when very loud noises occur, which is a form of low-level synaesthesia in itself). Reinforcing the connection, on some hallmark features of autism, such as attention to detail, synaesthetes effectively score as highly as those officially classed as autistic.

There are also key similarities between synaesthetic and autistic brains: both are characterised by abnormal connectivity between neurons and regions. Although there is some controversy about the nature of this different connectivity pattern, it generally seems as if both autistics and synaesthetes have hyper-connected brains, especially in how local areas are connected, and especially in complex information processing regions, compared to more "neurotypical" subjects 　. Such hyper-connected brains, especially in regions designed to process "higher-order" information, may help explain both how senses can be combined in both autism and synaesthesia, and how both conditions easily lend themselves to imbuing such experiences with deeper connections and rich meaning.

Synaesthesia boosting cognition?

Given this link between autism and synaesthesia, and given that a significant subset of autistic individuals express some savant skills, is there some performance benefit from being a synaesthete? Increasing evidence is emerging that synaesthetes have various advantages in perception (for example in visual or colour acuity) and memory 　.

But is this just a coincidence? Maybe, for example, synaesthetes just were born with better memories. One way to test this, I realised early on, was to attempt to turn non-synaesthetic adults into synaesthetes, by training them.

During this time, in 2010, I was in between my first and current stint at Cambridge University, having just joined the Sackler Centre for Consciousness Science at the University of Sussex (on the south coast of England). Within a year of my arrival, I was very lucky to have a colleague and soon-to-be friend join me from Switzerland, Nicolas Rothen. He had just published a paper trying to train synaesthesia himself, but the subjects never developed synaesthetic experiences. This was essentially in accord with the literature, where it didn’t seem possible. But a key difference between these studies and what happens in the real world to turn non-synaesthetic children into genuine synaesthetes is training time.

For instance, take the most common type of synaesthesia, grapheme-colour synaesthesia, where a visual letter or digit has a consistent colour related to it. This probably develops over the years that school children are most actively learning to read and use numbers, in other words the ages of 6 to 8. Returning back to my volunteers "cheating" on the squares short-term-memory task by seeing patterns in the sequence, I think a very similar process is happening in analogous fashion with some children as they are acquiring this tremendously difficult, abstract task. Perhaps because certain children are more inclined to focus on certain details of the world, or because they have creative strategies for dealing with memory challenges, or because of the pressure of struggling more than other kids initially in learning to read or use numbers or letters, they develop strategies to help them. The simplest strategy is to link such abstract symbols with something far more salient, like colours, especially if there is some semantic link between the two (such as "r" being red, a common synaesthetic association). So that’s what they do. Perhaps, in some circumstances, they are helped to make this link through simple childhood toys, like fridge letter and number magnets. Indeed, adult synaesthetic letter-colour associations can bear a striking resemblance to the details of such childhood toys. While these adults have no memory of the source of their associations, for someone like Shereshevsky, with his exceptional memory, he was indeed able to recall that the colours and even the precise font of his synaesthetic numbers originated exactly from the toys that taught him the digits as a child.

Can synaesthesia be trained?

Starting, possibly, with semantic seeds such as fridge letter magnets, in order to establish synaesthesia on a difficult abstract set of features to learn, you need years of daily focusing on such a task, whereas Nicolas’s earlier synaesthesia training study was only 10 days long. It simply wasn’t feasible for us to train people in the lab every day for years.

Our compromise was two months training for these initially non-synaesthetic volunteers, with a range of highly motivating tasks that changed week by week, and which constantly changed difficulty levels according to the abilities of the volunteer. For each of these tasks, a consistent colour was linked to one of thirteen letters. Amazingly, after these two months, many of the volunteers reported experiences indistinguishable from real synaesthetes, such as seeing colours superimposed on letters, outside of the lab, just as they were walking around the university campus 　. Performance improvement for some of the subjects, especially in the most demanding short-term memory tasks, was striking. There was even provisional evidence that their IQs significantly improved.

Given how novel these results were, we thought it important to replicate it, which we did a few years later, this time with the addition of brain-scanning components. This further demonstrated that after the training period not only were our previously non-synaesthetic subjects having synaesthetic experiences, but in addition their brains had adapted to respond to visual stimuli in ways that only genuine synaesthetes normally do 　.

I should emphasise that these results were only temporary, and three months post-training the effects had largely worn off. Perhaps had the subjects kept training for many more months, the situation would have been different, but that experiment has not been run yet.

Nevertheless, our training results suggest that the link between synaesthesia and improved performance isn’t a coincidence, and that acquiring synaesthesia itself causes cognitive benefits.

Based on these, we are keen to turn these results into clinical tools, based on apps, to help treat certain conditions, such as dyslexia and dyscalculia. If you would like to contribute to this research, you can donate here.

Synaesthesia as savantism?

One recent study from Jamie Ward’s lab tied all these strands neatly together with the following question: what would happen if you trained synaesthetes on a skill normally associated with autistic savants? For this study 　, they used synaesthetes whose days, months and years aren’t associated with a colour, but a specific spatial location. The skill they were trained on was related to this – that of calculating the day of the week from a specific year and date (one of the savant skills that the autistic synaesthete, FC, above, demonstrated). It was a skill they had no ability for before the training, but after a few weeks of training, the synaesthetes were able to perform this task significantly better than non-synaesthetic controls, who were learning the same tricks for the same time. In this single result there is striking overlap between the three fields of synaesthesia, autism and savantism.

The power of perceiving patterns

It’s increasingly clear that there are close links between synaesthesia, autism and increased performance in certain fields. But why should this be the case? The answer may lie in the power of noticing and using patterns, either in the world, or within our own minds.

An obvious truth is that the information of the world around us is not random. It follows rules – the laws of nature. These laws are complex, and multi-layered, but the more an animal can decipher those laws, the more it can flourish and thrive in its environment. As humans, we are a runaway success story in this regard, given we can not only discover very deep patterns of the world, but we can pass this knowledge on, and build on those foundations, floor by floor, with each generation. That, essentially, is what science does.

On an individual level, though, we can also benefit hugely from developing a deeper representation of the patterns around us. As humans, we are hungry to find such patterns. An autistic brain, with its hyper-connectivity, its obsessive attention to detail, and increased interest in pattern-based information, like calendars and music, is perhaps more likely than others to reap the rewards of this in savant abilities. It is consequently no coincidence that many scientists appear on the autistic spectrum.

Synaesthesia, long assumed to be an independent condition to autism, bears many similarities to it, both behaviorally and neurally, such as changed brain connectivity, and a propensity to obsess over specific features. Whether this is to compensate for some initial deficit (such as struggling to learn to read in childhood), or merely reflects a similar increased propensity to superimpose patterns on the information we internally process, is an open question. But it’s clear that synaesthetes, like autistics, are hyper-sensitive to various sensory inputs, and can often exploit the extra structure they have from their synaesthetic percepts to gain performance advantages.

One key point here, though, is that this doesn’t happen, especially initially, unconsciously. Instead, consciously perceiving stimuli seems key to synaesthetes gaining benefit from their extra mental patterns 　. The same story is played out in psychology again and again: for any form of complex learning, especially involving deep multi-level patterns, consciousness is required. It is such a powerful learning tool that this ability to gain deep insights might be the key purpose to consciousness itself, as I’ve written about elsewhere. Autism and synaesthesia, in their complementary ways, demonstrate the incredible variety and creativity our conscious minds possess to discover and perceive structure in the world around us. And when we exercise this skill adeptly, we reap potent rewards in terms of the exquisitely deep understanding and mental gymnastics that we can gain from this structured mental landscape.