The algorithms of art

CHRIS HATZIS
Eavesdrop on Experts, a podcast about stories of inspiration and insights. It’s where expert types obsess, confess and profess. I’m Chris Hatzis, let’s eavesdrop on experts changing the world - one lecture, one experiment, one interview at a time.

Can mathematics be described as an art form? Philosopher Bertrand Russell once said:

“Mathematics, rightly viewed, possesses not only truth, but supreme beauty...”

Kate Smith-Miles is a Professor of Applied Mathematics and Chief Investigator with the ARC Centre of Mathematical and Statistical Frontiers, University of Melbourne.

Kate's research quest to stress-test optimisation algorithms led to a large collection of intricate and beautiful 2D images, contour plots of mathematical functions that’ve been mathematically generated to create challenging landscapes.

Kate recently presented a webinar titled “When Mathematics Becomes Art: the unexpected beauty of self-evolving mathematical functions.”

Professor Kate Smith-Miles sat down for a Zoom chat about her work with Dr Andi Horvath.

ANDI HORVATH
Kate, what's your research about?

KATE SMITH-MILES
I'm a mathematician by training but, lately, I've started to become very interested in how mathematics can help us develop trust in algorithms, because algorithms are everywhere, and it's becoming a really pressing issue for how we can trust them. The good news is that mathematics and statistics offer some really valuable tools for us to be able to develop algorithmic trust.

ANDI HORVATH
Now, algorithms are like strange little creatures that drive our computer programs, but why is it that we're not in control of algorithms? Why are they such a messy thing?

KATE SMITH-MILES
Well, I suppose algorithms are black boxes for many people. We don't really have a good understanding of what you feed the algorithm in, and what it's going to do as output. The mystery of what's inside that black box, the programmer knows, but if we're going to trust algorithms, we're going to rely on what they tell us, and we're going to trust those results. We either get some insight into what's inside the black box, which is beyond comprehension for most people, or we trust that they've been tested really thoroughly.

ANDI HORVATH
Can you give us an example of an algorithm that's gone horribly wrong?

KATE SMITH-MILES
There've been quite a few examples in the media in recent years, where algorithms for – in particular, things like face recognition, have been accused of being biased or even racist, where the algorithm has given an incorrect response – for instance, there was a young African-American man in a New York department store, who was arrested under suspicion of theft, because a face recognition algorithm in the store's security camera incorrectly identified him as somebody that was wanted. When they investigated - there was a big senate inquiry in the US into racial bias in some of these algorithms. That's just an example, where we know that algorithms are not necessarily tested in a way that can guarantee that they're not biased. It's the classic case of typically white men developing algorithms that work particularly well for white men, but for minority groups, they're usually not tested rigorously enough.

ANDI HORVATH
It sounds like we kind of need to optimise our testing and design of algorithms. Is that what you do?

KATE SMITH-MILES
Yes. I look carefully at how we can choose the test problems that we throw at algorithms, to try to understand whether they're performing reliably. It's not just a question of giving an algorithm enough – it's not a quantity question, it's the quality of the test examples. Most important is the notion of diversity. We need to be able to be mathematically sure that we're giving an algorithm the right kind of test problems; not just enough of them, but that they have enough diversity that we can expose all of the weaknesses that might be hidden.

ANDI HORVATH
It almost sounds like you need to put ethics and social sciences into the design of algorithms.

KATE SMITH-MILES
Yes. It's a very multidisciplinary field. It's because it's such a critically important area that's going to impact our society, and we need all the best brains to be thinking about it from all perspectives.

ANDI HORVATH
I recently attended a talk you gave, Kate, entitled “When mathematics becomes art: the unexpected beauty of self-evolving mathematical functions.” How did that start?

KATE SMITH-MILES
This was a very unexpected outcome of some mathematics research. My area of research in mathematics is optimisation. I'm trying to solve optimisation problems. Optimisation problems arise in all parts of life where we're trying to maximise or minimise something, whether it's maximising profit or minimising traffic congestion – lots of different applications.
I became interested, first of all, in how we test optimisation algorithms. The thing is, as an academic, we have a research culture and a training, where we test algorithms by downloading from a website the commonly studied test problems. If your algorithm performs well on these commonly studied benchmarks, then you can publish your paper. It's kind of like a rule of thumb.

I've been an editor of journals, where I got a little bit tired of having papers submitted to the journal that were along these lines: Here's my new algorithm, and I tested it on these 10 problems, and I got .001 per cent better than everybody else, so please publish my paper; it's really good. It's like, yeah, but tell us about those problems. Why did you choose those 10 problems? Were you just kind of cherry picking, and choosing problems where your algorithm performed well but you've ignored or overlooked, or not shared with us any of the other problems? I started to realise the importance of how we can rigorously test and be convinced that somebody isn't just shining a positive light on their algorithm.

I started to question the choice of test problems, and how we could develop an understanding of the most comprehensive set of test instances; what this would actually look like - not just the quantity, but the quality of them.

I developed a methodology, underpinned by mathematics, and then I started applying it to my original field, to optimisation algorithms. What we discovered was that we can generate a whole bunch of new test problems that will guide future testing of algorithms. But it turns out that these optimisation problems that we generated to have increased diversity; they're really quite fundamentally different from some of the other test problems that everybody uses.

The research took a direction where we're trying to generate – deliberately generate diverse problems – unique problems. It turns out, we were able to visualise them as 2D, beautiful images. We had so many of these beautiful images, that it turned into a new motivation to try to create an artwork. That's what I spoke about at the lecture last week.

ANDI HORVATH
So you created kind of like a patchwork quilt of these various – let me describe them as sort of organised tie-dye variations on a theme. These pictures are kind of like a patchwork quilt of variations of organised tie-dye images of yellows, greens and blues. Tell us what happened next.

KATE SMITH-MILES
That's a fantastic description of them. I tend to describe them – some people call it like paisley patterns. They see paisley patterns. Other people – mathematicians often ask me are they fractals?

What happened next was we had 306 beautiful images, and I wanted to print one for my office, but it's like choosing your favourite child – I really couldn't choose just one. So I started asking people which one do you think is nicest? And what does nicest mean? If you start thinking about these aesthetic properties, what is the most aesthetically pleasing image? It quickly became clear that I would not be able to choose one, so we started arranging them in a montage, or an array, or a patchwork quilt, as you described it. So 17 x 18 is 306. So we have a 17 x 18 array of little images. Then the question becomes what is the most aesthetic arrangement of these images? Because what we found is that when we randomly place some of these images together, a lot of the background blue structure starts to dominate if there's too many images next to each other that have sort of a blue continuity through them. The eye was drawn to that region of the patchwork quilt. So it became a very interesting mathematical question of what is the optimal arrangement of these images to maximise aesthetic appeal?



ANDI HORVATH
So you got your colleagues to vote on which patchwork quilt actually soothed their soul, so to speak. I heard you describe them as the pattern of the blue rivers that they saw in amongst this patchwork quilt. What happened next? Because you went into the science of aesthetics even more deeply.

KATE SMITH-MILES
Yes, I suppose as researcher you turn to the literature, and when you observe something you want to know am I just crazy or is this a thing? It turns out, it is a thing, that different people will find different things aesthetically appealing.

I observed, when I surveyed my friends and family, I posted things on Facebook, and asked people to tell me which patchwork quilt they found most appealing. I started to observe some patterns of preference for, I'll say, order versus disorder, and it aligned quite interestingly with people's personalities. So people who tended to like a lot of order and structure in their life, very much strongly expressed preference for the arrangement of the images, where there was a pattern that they could see of how the images connected to each other. It didn't look random; it had a structure. In fact, it was the dark blue connectivity that – it created these sort of meandering rivers all through the patchwork quilt. Whereas some people didn't like that at all. They weren't wanting that. They found that to be a distraction, in fact. They wanted to appreciate the pure randomness of the diversity of these beautiful images. All they needed to make sense of it, was the array structure itself.

In fact, my daughter, who's – she's nearly 18, very artistic, creative type, she expressed a very strong dislike for the imposition of order. She didn't think that I should be trying to connect the blue rivers, because that's what I did. What I found was that people didn't actually like the frustration of a random arrangement that had a little bit of partial structure there, some blue rivers started to form, but then they would just abruptly stop. So, if you were looking for structure, you found that frustrating. If you didn't want to see any distraction of blue rivers, you found it annoying that they were there to start with. So I formed the opinion that people actually don't like random, they want me to rearrange these images to either get rid of any sense of structure at all, or to enhance it.

So I set about with a new scientific objective, to try to get an algorithm to do this. At the same time, I had an artistic goal of, actually, I think I can just do this with my eye. It turns out that the artistic effort was quite pleasing, people either liked the one with more order or less order. Our efforts since, to try to get an algorithm to do the same thing, have been a little bit frustrating, I have to say.

ANDI HORVATH
That's really interesting, because that kind of says the algorithm doesn't do what our human pattern-seeking brains, and the variation of the human pattern-seeking brains, like your daughter, who prefers a more randomness to others, who preferred more shapes, that they kind of liked to flow with, is an extraordinary revelation. Our computers and algorithms cannot match our pattern-seeking notion of aesthetics. I find this really exciting. So this is, essentially, described aesthetics from a mathematics point of view.

KATE SMITH-MILES
Yes. It's been quite interesting. Once I had this observation, that my friends' personality seems to correlate quite nicely with their aesthetic taste, I became curious about whether this is actually known already or am I onto something? Turns out, I'm not onto something, this has been known for a very long time.

In my lecture, the other day, I whizzed people through two-and-a-half thousand years of aesthetics, understanding the history, philosophy and science of beauty and aesthetics.
But, certainly, in the last 50 years or so, there's been a lot of progress made in understanding how individual preferences affect aesthetic taste. Lately, a field has emerged called neuroaesthetics, where we can understand through CAT scans, and things, we can see what's happening in different people's brains when they're exposed to different visual stimuli.

Before that, there was lots of research done in how individual personality preferences might affect your taste. The big five psychological personality traits, like neuroticism, and openness to new experiences, and these kinds of things, people can be pigeon-holed into different personality categories. There's been a lot of research in how that affects their aesthetic taste and judgement.

ANDI HORVATH
I love it. Neuroaesthetics – I'm going to use that word in a sentence tomorrow. Now, tell us about yourself, Kate. You've been a mathematician pretty much all your life. What got you into this area of mathematics? How did it all start?

KATE SMITH-MILES
Well, I was very lucky. In my final year of high school, I had a wonderful mathematics teacher. Before that, I was always good at maths, but I didn't love it. It wasn't something I was considering as a career path. During Year 12, I had a mathematics teacher who really helped me understand some beautiful things about mathematics, and it was just scratching the surface. I got to the end of Year 12, and I thought, no, that can't be the end of it; I have to keep going with this because I've just got a little glimpse of what this might actually be.

The stuff we learn in high school is not real maths, it's foundational stuff. I always say to people, it's like learning grammar versus writing great works of literature or learning to play musical scales versus playing an actual masterpiece. In mathematics, the stuff you learn at high school is just the foundations, and it can be a little bit dry and dull. But I got a glimpse towards the end of Year 12 of, no, it's not what I thought it was; there's more to this, and I wanted to keep learning.

So I decided to go to university to study mathematics, without knowing where it would take me. But I figured, they wouldn't teach it if it didn't lead to something. So I took the gamble.

ANDI HORVATH
What sort of misconceptions have you encountered that people have about mathematics in your area?

KATE SMITH-MILES
I think there are so many misconceptions about mathematics. Not helped by the Hollywood stereotype. Unfortunately, we have too many movies that have been made about some mathematician in trouble, in their life – mental illness. Mathematicians are usually portrayed as some lonely, brilliant person that's a little bit disconnected from the real world. So it's not an attractive aspiration for most young people. I think we battle against that.

When I tell people I'm a mathematician, sometimes they're a bit surprised. They don't think that many women are interested in mathematics. There is a lot of stereotypes around whether female brains are different to male brains and whether you have a maths brain or not, which is all just rubbish. There's been so much research that has demonstrated that that's a genuine misunderstanding that people have. There are plenty of men that don't have maths brains as well. The notion of a maths brain itself is a bit fraught. Everybody can be taught to do mathematics.

It's one of my frustrations, that we have a tendency in society to think it's okay to not be good at maths. The number of people who say, Oh, I was hopeless at maths at high school, and I've lived my life just fine without it. We would be horrified if somebody went around, saying, Oh, I'm hopeless at reading. People don't admit to that. Yet, people are quite happy to admit to being hopeless at maths.

Yet, developing a mathematical way of thinking, it's like brain training, it's developing a logical, rational way of thinking that mathematics training gives you. It's really important that everybody be given an opportunity to develop those skills but, far too often, people dismiss it as being optional.

ANDI HORVATH
Mathematics is kind of a universal language. It's kind of like music in that sense.



KATE SMITH-MILES
Yeah, mathematics is a language, and it's an important language that everybody needs to learn to speak. Certainly, for the future generations, they must be able to converse in mathematics. And I don't just mean mathematics as in some equation or some algebra, I mean logical, rational argument. That's what mathematics gives us. It gives us the foundation for how we set about logically arguing, stating our assumptions, and all the things that follow from those assumptions in a logical sequence.

ANDI HORVATH
Kate, next time we see equations on a whiteboard, or just equations and numbers, what would you like us to think about?



KATE SMITH-MILES
I think instead of people immediately being fearful or disinterested, I would love it if people could see that equation and just imagine the possibilities of what that equation could be describing. Because, as you said, mathematics is a language, and equations describe things in our world.

I'll always remember, when I was in third year at university, and a lecturer wrote an equation on the board, and he said, as a throwaway line, this equation can be used for modelling traffic flow. I thought, wow. I didn't realise that, that we were learning stuff that was so immediately applicable to the real world. So, yeah, equations are such an important part of a mathematician's toolkit for looking at the world, figuring out what it's doing, and how we can improve it.

ANDI HORVATH
Professor Kate Smith-Miles, thank you.



KATE SMITH-MILES
My pleasure.



CHRIS HATZIS
Thank you to Kate Smith-Miles, Professor of Applied Mathematics at the University of Melbourne. And thanks to Dr Andi Horvath.

Eavesdrop on Experts - stories of inspiration and insights - was made possible by the University of Melbourne. This episode was recorded on October 13, 2020. You’ll find a full transcript on the Pursuit website. Production, audio engineering and editing by me, Chris Hatzis. Co-production - Silvi Vann-Wall and Dr Andi Horvath. Eavesdrop on Experts is licensed under Creative Commons, Copyright 2020, The University of Melbourne. If you enjoyed this episode, review us on Apple Podcasts and check out the rest of the Eavesdrop episodes in our archive. I’m Chris Hatzis. Join us again next time for another Eavesdrop on Experts.