On Painting and Algorithms

Eric Joyner

The following is an essay I wrote around 2010 about painting and computers. I stumbled across it recently as I was looking through my notes in preparation for my next blog post. It reminded me that tech developments back then, which now seem comically quaint in relation to AI, were raising many of the same questions and concerns artists have today. It's interesting to see the seeds of AI already being planted with the rise of supercomputers (remember Watson?) art filters ("turn your photo into an Impressionist painting!") and various other emulation programs. Enjoy the trip down memory lane:

---

On Painting and Algorithms

If you wanted to paint an apple in the manner of Cezanne, you would need to have extensive knowledge of his work and a strong sense of his aesthetic intentions. Every brushstroke would need to be accompanied by the question, “What would Cezanne do?” You might stop frequently to refer to his paintings to see how he handled certain visual situations. As your painting progressed you would gradually develop perhaps a dozen general stylistic guidelines for yourself. These guidelines would be instructions along the lines of “When you see this, do this.” Of course, much of the process would be based on wordless intuition; a vague sense of when a group of marks looked “Cezanne-esque.”

On a basic level, this is not unlike the way a computer algorithm works. An algorithm is "a finite sequence of instructions, logic, an explicit, step-by-step procedure for solving a problem, often used for calculation and data processing and many other fields." An algorithm acts as a kind of flow chart which guides a computer through a series of evaluations and decisions. When translating, say, a Shakespeare sonnet from one language to another, a computer will use an algorithm to evaluate and substitute words and phrases into the other language. This is called “gisting” because computers are still not capable of making a translation that is much more than 80% accurate—for all their processing power, computers have a difficult time processing the complexities and nuances of contextual meaning. In the art of literary translation, there are no clear-cut right or wrong rules for choosing a phrase that means the same thing in one language as it does in another and that keeps the same rhythmic or emotional characteristics. From Wikipedia on the art of translation:

Fidelity (or faithfulness) and transparency are two qualities that, for millennia, have been regarded as ideals to be striven for in translation, particularly literary translation. These two ideals are often at odds. Thus a 17th-century French critic coined the phrase les belles infidèles to suggest that translations, like women, could be either faithful or beautiful, but not both at the same time.

Here is a Shakespeare sonnet translated from German into English by a computer program:

I am to compare one summer day you, which you

lovelier and moderate are? Mays expensive buds

drehn in the impact of the storm, and is all too short

summer period.

You get a general idea of what the words mean but the poetry is missing. The best a computer can strive for is faithful. Beautiful, for the time being, is out of the question.

In painting a Cezannesque apple you would, in essence, be acting as a kind of translator. Specifically, you would be trying to translate one visual language (Nature’s) into another (Cezanne’s.) Or, in Photoshop parlance, you would be acting as a Cezanne filter.

There are no Cezanne filters that I am aware of, but there are, of course, the increasingly ubiquitous art filters that use algorithms to manipulate or imitate a film type or fine art media. Interestingly, in the 1800s a number of photographers attempted to make their photos look more like paintings by manipulating the development process to achieve painterly effects— an analog version of an algorithmic photo filter. Computer applications can be quite sophisticated in terms of their ability to mimic natural media like oil paint or watercolor—both in terms of their appearance and their working properties. There are also more and more programs now that attempt to mimic specific styles of painting like, say, Impressionism or Pointillism. You feed the program a photo and it spits out an "impressionistic" version of it. The results are almost always very bad. Like language translation, art filters are far too simplistic to handle the contextual/aesthetic complexities of painting. It is hard enough for a human to define what a good painting is, much less write an algorithm that can define it for an unthinking computer.

But computers are getting more powerful and algorithms more sophisticated. Massive databases of information can now be accessed so quickly, and patterns discerned so efficiently, that a computer can appear to be sentient, as when the IBM supercomputer "Watson" competed against two Jeopardy champions. The computer had access to 200 million pages of information that consisted of raw data like encyclopedias and dictionaries, books, news, movie scripts, etc. It was not connected to the internet or guided by any human helpers. As with most computers, Watson’s weakness is the inability to understand the nuances of speech and language or to have any life experiences to draw upon to divine answers. Scientists alleviated these problems by loading the data onto the computer’s RAM rather than the hard drive, which made searches much more quick and nimble. Algorithms were then designed to take advantage of this increased speed to find subtle patterns and probabilities inside the mountain of data. So Watson listened to Alex Trebeck, rang the buzzer, and answered in the form of a question, all without human intervention. Watson won.

Another interesting example is a program called “Emmy” designed by David Cope, a professor at UC Santa Cruz. Cope was having trouble finishing an opera commission so he designed a program that could emulate the work of several great composers to help spur his thinking. Emmy uses an algorithm to find patterns in a great composer’s music and then uses that information to piece together the composer’s style and create a new composition. When an audience was asked to listen to an Emmy-created Bach composition and a real Bach composition, they could not tell the difference. One could argue that the “new” compositions are merely derivative and so not new at all, but couldn’t the same be said of human composers? As Picasso once said, “Good artists copy, great artists steal.”

At the University of Georgia, Gil Weinberg designed a robot, named Shimon, that can interact with other musicians and also, supposedly, play and improvise like Thelonious Monk.

He says that, though he and his team were trying to teach the robot to play like a machine, they first had to teach it how a human plays. To do that, they used statistics and analysis of Monk's improvisation. Once they had a statistical model of the pianist, they could program the robot to improvise in that model. Weinberg says the robot won't play everything exactly like the bebop pianist—or any other jazz master—would, though he says, "It probably will keep the nature and the character of [the musician's] style. It's difficult to predict exactly what they would do in every single moment in time," he says. "But our algorithm pretty much looks at the past several notes that it plays and, based on that, it sees what is the probability of the next note to be, based on all of this analysis of a large corpus of transcribed improvisation." (NPR, 2009)

I think it is likely that in the near future, there will be a painterly version of Emmy, Watson, or Shimon that can digitally paint in the manner of an artist by analyzing an enormous database of that artist’s work and perhaps even the work of those who influenced him. The output will vary in quality, of course, and depend a great deal on the appropriateness of the input, but I’d guess that at least some of the resulting images will be quite convincing. Also, it is not hard to imagine that in five or ten years, display screens will be capable of displaying images that, from a few feet away, are virtually indistinguishable from real paintings. Perhaps they will be like the Kindle screen, except able to reproduce millions (billions?) of colors and have a resolution that not only reproduces the details (think of the Google Art Project) of the depicted image but also accurately conveys the texture, sheen, and depth of the brushstrokes. This screen would likely be very thin, light, and easy to hang on a wall. It would also be fairly inexpensive and have a battery life of months rather than hours or days. And, as an aside, maybe there will be a company called “Artflix” rather than “Netflix” whereby one could download an ultra-high resolution image of a great painting (I suppose the company would have to work out some kind of revenue-sharing system with the museums and galleries that owned the rights to those paintings—as iTunes did with the music labels.) You could rent a Vermeer for a week.

We all act as filters to some degree. Our minds edit incoming signals (photons don't have color, for instance—we assign them colors via our brain.) These are primal algorithms over which we have very little control (we do not have a choice to see in black and white.) The human algorithms I am referring to are those decision matrices we use by choice in the course of a painting—the processes, techniques, and methods that we learned in our training and practice. In one sense they help us build our paintings by freeing us up to focus on the larger idea of what we want to express. In a painter's formative years, he borrows algorithms because emulation is one way a painter learns from other painters. Borrowed algorithms serve as temporary bridges that allow him to cross artistic waters he may not have the experience or knowledge to navigate alone. As he practices he slowly develops his own algorithms. However, an algorithm can devolve into a habit of sorts, if a habit is defined as an automatic reaction to a specific situation. Painters are tempted to rely on such habits because they allow them to avoid the risk inherent in painting, and thus mitigate the struggle. The quest for a technique or method often turns into a quest for shortcuts—that is, the successful deployment of a technique or style becomes an end in itself. The larger thought or idea that style or method was supposed to serve gets lost in the pursuit of risk avoidance and efficiency.

A kind of analog version of an algorithmic filter is the paint-by-numbers painting system. In paint-by-numbers, one is presented with an image divided up into numbered sections, with each number assigned a color. You paint each section with the corresponding color. If one designed a PBN system using thousands of colors and thousands of sections and perhaps added other parameters like degrees of softness to edges or types of brushstrokes (thick or thin, fat or slow, etc), the end result might appear to be quite sophisticated and intricate. Add a few more subtle variations and a painter could come to believe he was following his own muse rather than a set of instructions. The whole purpose of paint-by-numbers is to make painting a pleasurable, soothing experience like putting together a jigsaw puzzle. It takes patience and some skill but your path is made plain and you know what the results are going to be beforehand. The painting is a foregone conclusion, no matter its complexity, and the smell of paint belies the fact that the painter is simply a computer running an algorithm.

An example of efficiency in painting taken to an extreme can be found in the art factories of China. Sixty percent of the world’s mass-produced, cheap oil painting copies come from one small town (1.5 square miles) in China, called Dafen. A worker there can produce a couple of dozen copies a day by hand and it is estimated that 5 million paintings are produced in Dafen every year. There are assembly lines, too:

Dafen—and other villages like it—are bringing the factory assembly line into the artist's studio. In a dimly lit hall on the outskirts of Dafen, “painter workers” stand side by side dabbing colors onto canvas. Liu Chang Zhen, a 27-year-old, works eight hours a day to complete more than 200 canvases a month—painting several copies of a picture at a time, methodically filling in the same patch on each before moving to a new part. At other factories, painters work on the same product but specialize in different parts—in ears or hands or trees. They work from art books, postcards, and images from the internet. Sometimes they just paint inside an outline copied electronically from a photograph, enlarged, and stamped on the blank canvas. (The Economist)

These workers are trained to be, first and foremost, efficient. They find the quickest, easiest way to complete a technique so that it can be repeated without much thought. Apparently, there is little pretense among the workers that this is high art, but workers do take pride in the specific skills required. In Dafen, for instance, there are regular art competitions where several dozen workers compete to see who can complete a copy (or a “replica” as they are referred to) of a masterpiece the fastest and most accurately. It is art as sport.

I am sure that one-day robots, using algorithms and printers (or perhaps using real brushes and paints) will replace these assembly line workers just as robots replaced many workers in industrial factories here. Looking at the medical robot, Da Vinci, it is not hard to imagine it manipulating a brush. Low-level, repetitive jobs are always assumed to be the ones that technology targets and replaces first, but "low-level" is being defined up:

Tuesday was a great day for W. Roberts, as the junior pitcher threw a perfect game to carry Virginia to a 2-0 victory over George Washington at Davenport Field. Twenty-seven Colonials came to the plate and the Virginia pitcher vanquished them all, pitching a perfect game. He struck out 10 batters while recording his momentous feat. Roberts got Ryan Thomas to ground out for the final out of the game. Tom Gately came up short on the rubber for the Colonials, recording a loss. He went three innings, walked two, struck out one, and allowed two runs. The Cavaliers went up for good in the fourth, scoring two runs on a fielder's choice and a balk. NPR April 17, 2011

The above excerpt was written by a computer program that writes local sports stories using the statistics from the game as its only source.

My reason for discussing such technology is not to sound an alarm about computers replacing artists. The computer will not make painting obsolete any more than photography did, but I do believe it will be disruptive. Much of what we see now in terms of painting and computers is in its infancy and, like most technology, when it first starts out it can appear simplistic or silly. However, it did not take long for photography to become the de facto way to record visual facts, and as cameras grew smaller, cheaper, and more efficient it became evident: if your job as a painter was to merely paint facts, your job was threatened by a box with a pinhole in it. Similarly, now, if your mission as a painter is to blindly follow a set of visual rules (“when you see this, paint this”) then your equal will soon be a piece of silicon. Photography started an ongoing conversation about what painting is and what it should be and I believe computers will soon rekindle this conversation.

Duane Keiser, 2010

Read my articles on AI and painting here