There is something funny about art and graphic design that seems to elude any rule or system you try to apply to it. Indeed, it is perhaps the nature of art to transcend any rule. Given the number of words and paragraphs written about it, trying to define it, I have no illusions that a ruleset can be found that clearly and concisely defines art in terms of language, or better yet, programming language.
But there is a strong resemblance between what happens in a work of art and what is being done in the field of Artifical Intelligence. A work of art is too complex to extrapolate to a set of rules: it involves composition, geometrical and contrast factors, but also the conceptual and contextual. A work of art is not just a visual thing, but something that relates and connotates to the world outside, the world of the artist, and the world of the viewer. Quantumwise, a work of art is only "fixed" when it is looked at, and it has a different state or meaning for each different onlooker. In short, there are simply to many factors to box art (in for example programming code).
How then is it possible to create any visual work that captivates the attention, that is beautiful to look at, that leaves a profound sense of meaning at all? If the artist doesn't know the art-ruleset, how is he able to create it? Some hidden subconscious instinct? I think not. In Artificial Intelligence there is something called emergence: the formation of complex patterns from simple rules. An infinite number of simple neurons together make up a powerful brain capable of thought. A single ant has a limited set of actions and capabilities, but as a colony, they are intelligent, able to solve geometric math and organise huge social structures. They are self-organising as a system, though alone they have no clue what is happening, act at random.
"An emergent behaviour or emergent property can appear when a number of simple entities (agents) operate in an environment, forming more complex behaviours as a collective. The property itself is often unpredictable and unprecedented, and represents a new level of the system's evolution. The complex behaviour or properties are not a property of any single such entity, nor can they easily be predicted or deduced from behaviour in the lower-level entities." - Wikipedia
It is my opinion that art has an emergent property as well, the magic of seeing all elements come together, fit together, work, while at first everything on the canvas looked puny and silly. The power of art and graphic design lies in the relationship between the visual elements, not the elements themselves. The complete oevre of an artist tells a story, while individual works might be ugly or strange.
"Thus it is not just the sheer number of connections between components which encourages emergence; it is also how these connections are organised. A hierarchical organisation is one example which can generate emergent behaviour (a bureaucracy may behave in a way quite different to that of the individual humans in that bureaucracy); but perhaps more interestingly, emergent behaviour can also arise from more decentralized organisational structures, such as a marketplace. In some cases, the system has to reach a combined threshold of diversity, organisation, and connectivity before emergent behaviour appears." - Wikipedia
Ant Colony Optimisation is a computer model of an ant hill, able to find optimal solutions for the travelling salesman problem, something that was thought to be more or less unsolvable due to the number of parameters involved. Perhaps with art as well, the number of parameters and influences is too high too find any ruleset that defines it. But really simple rules that have emergent behaviour could provide an answer here.
For example, we have been dabbling with composition a long time now, finding no good computer algorithm to simulate composition in art. We were trying to take into account all factors and parameters. But recently, moving to flocking algorithms (each "bird" or "fish" has a few simple rules but as a flock they work together), produced far better results: the elements found the composition by themselves, and almost all of them look harmonious. Emergent behaviour.
Another example is Gravital, a design language that defines itself; it emerges from itself and is in constant growth - no single person has to define all the rules of art by itself.
Ants transporting food to the hive: