Before the topic of artificial intelligence sparked global debate in recent months, two university professors in Hong Kong had already been thinking about the various potentials of AI creation.
Guo Yike and Johnny Poon, who are responsible for coordinating the "Hong Kong Human-Machine Symbiotic Art Creativity Platform" project at Hong Kong Baptist University (HKBU), led the team to create the world's first human-machine co-creation and co-performance song and dance performance at the end of last year. It combined an AI choir, AI dance, AI media art, and a live symphony orchestra to perform at the HKBU Anniversary Concert.
"Hong Kong Commercial Daily" interviewed the two professors to explore their research fields and glimpse the future of human-machine co-creation of emotional artworks.
NO.1 AI Art: Human-Machine Co-creation of Emotional Works
The virtual choir developed by Hong Kong Baptist University's "Hong Kong Human-Machine Symbiotic Art Creativity Platform" has four voice parts.
HKBU's "Hong Kong Human-Machine Symbiotic Art Creativity Platform" project received a grant of over HK$50 million from the Research Grants Council of the University Grants Committee, and started a 5-year special research project in 2021.
As project coordinators, the two interviewees themselves also present a combination of art and technology. Guo Yike is the founding director of the Data Science Institute at Imperial College London, currently the Provost of the Hong Kong University of Science and Technology (HKUST), and teaches in the Department of Computer Science and Engineering. Johnny Poon is the founding dean and professional conductor of the School of Creative Arts at HKBU, and currently serves as the Associate Vice-President (Interdisciplinary Research) of HKBU.
HKUST Provost Guo Yike (right) and HKBU Associate Vice-President (Interdisciplinary Research) Johnny Poon (left) jointly study the infinite possibilities of AI creating art.
Professor Guo describes himself and Professor Poon as "a scientist who appreciates art" and "an artist who loves science." Guo and Poon worked together at HKBU years ago. Because both were deeply interested in AI art, they hit it off, which became the origin of the project.
Professor Guo pointed out that the ultimate goal of the research is to change existing art forms, not only integrating AI into art but also letting AI express humanity: "Any scientific research is driven by curiosity. The project wants to challenge how to make machines express humanity, and the team believes that experiencing art is the best way for machines to understand humanity."
Whether AI can generate consciousness is still an unsolved mystery. How to define letting AI express "humanity"? As an artist, Johnny Poon said: "If the work generated by AI can move me, I think the project has met the standard."
Human Conducting AI Choir
The human-machine song and dance performance performed at the HKBU Anniversary Concert last year was the first result of the research project. Johnny Poon conducted the AI choir and the live symphony orchestra at the same time, performing the newly arranged "Pearl of the Orient", accompanied by AI-generated ballet and visual art images on the large screen on the stage.
The AI trained by the research team can convert natural landscape images and music into dance images. Photo provided by Hong Kong Baptist University
The four-part AI choir was developed by HKBU's "Creative Intelligence Laboratory". By analyzing the singing recordings of professional singers, it extracts the "components" of vocal singing and creates a singing voice generation model, endowing the AI singer with a brand new timbre. The model can flexibly adjust the intonation, melody, and artistic expression of the AI singer, allowing the AI choir to collaborate with any orchestra in real-time.
Johnny Poon said that conducting an AI choir is like conducting real people. The AI singer can accurately identify his conducting movements, change the speed of the melody, the strength of the singing voice, etc., but what surprised him most was that the AI singer has a humanized interpretation: "When I heard the AI singing for the first time, I found that it actually had a personal style and its own set of understanding of the cadence of the song. In an instant, it was like a real person, making me ask: who taught it to sing?"
Guo Yike was also moved by AI creativity: "Accompanying the AI choir is AI cross-media visual storytelling. This model generates images consistent with the artistic conception of the music by analyzing the melody and lyrics. In the early stage of creation, the team first let the model analyze "Pearl of the Orient". When the music played, the images generated by AI changed from clouds to Bauhinia flowers. The beautiful pictures were unexpected. It was unexpected for us that AI could have such associative power."
He told the author that the team is preparing a new AI choir project, expected to be performed within this year, "This performance will reinterpret Vivaldi's "The Four Seasons", integrating many AI-generated contents and new technologies."
The Core of Symbiosis: Human-Machine Closed-Loop System
Since AI has personal style creativity and can create unexpected works, does it still need to coexist with humans?
Guo Yike believes that AI will become a new medium for human artistic creation rather than an individual separated from human creation. The project also aims to study how humans and machines can complement each other and coexist, pushing art to a new realm.
The "human-machine closed-loop system" studied by the team is the core of the project. Guo Yike explained that the system is people-oriented. Humans are responsible for training, testing, and adjusting the AI model to help AI learn. Humans then provide feedback on the works generated by AI, allowing AI to correct, modify, and create works that are more in line with human ideas. Humans and AI constantly interact and improve, forming a closed-loop model.
Johnny Poon explained: "For example, the team wants AI to create images based on lyrics, but the results generated by AI for the first time failed to reflect the imagery of the lyrics. The team will then train AI to learn the history and culture behind the lyrics through human analysis and labeling, so that AI improves with every cycle, the generated images become richer and richer, and finally achieve the result in the team's mind."
The "human-machine closed-loop system" will be applied to the next research project - "Empathy AI". As the name suggests, the technology will use AI to detect and respond to human emotions. "The next step is to know, on the basis of this closed-loop system, can human emotions and inspiration during creation enrich AI's creation?" Johnny Poon said enthusiastically: "I asked Professor Guo, is it possible for this technology to present the pictures in the human mind? He replied that this is not a question of 'whether it will happen', but 'when it will happen'."
Today's common generative AI are all language models, that is, people input keywords to AI and let AI generate content. Guo Yike said that the team is studying how to replace text instructions with emotional data: "The AI technology currently under study can already identify some basic emotions, such as anxiety, happiness, grief, etc. This can not only makes AI works better reflect human emotions but also train AI to judge beauty and ugliness and create more attractive works."
Johnny Poon puts on a sensor and conducts a live symphony orchestra and an AI choir at the concert at the same time. Photo provided by Hong Kong Baptist University
Johnny Poon continued to point out that the operating logic of Empathy AI is similar to that of language model AI: "People are sometimes moved by music. The team will use instruments to record and label people's physical signals under moved emotions, let AI identify stimulating factors and human reactions, such as what intervals and tonality make people feel happy, and learn the laws that move people, thereby becoming a 'sensible' AI. Or create music with different feelings such as cheerfulness, melancholy, and passion based on other human emotions."
The results of the Empathy AI project will also be displayed in the form of music performances. Unlike the first performance last year, this concert will be a three-party collaboration experiment between performers, audiences, and machines. The two professors revealed that in addition to AI performance art works created based on human body data, Empathy AI will also inject real-time emotional data of the audience present into the real-time performance, provide feedback on the spot, and generate new artworks.
Johnny Poon puts on a sensor and conducts a live symphony orchestra and an AI choir at the concert at the same time. Photo provided by Hong Kong Baptist University
As for how to collect audience emotional data, Guo Yike said there are many ways: "In addition to MRI that can observe brain activity, there are also lighter and easier-to-carry instruments, such as smartwatches that can measure heart rate and breathing; brain-computer interface chips newly developed by tech business tycoon Musk; electronic skin that can sense body temperature, metabolism, and hormonal changes, etc. These technologies can be used to collect physical signals from the live audience and help infer their psychological state."
He continued to point out that technology is developing rapidly today, and the remaining three years of the project do not rule out the development of new instruments that can detect human emotions.
Everyone Can Create with AI
Guo Yike believes that developing AI art will not necessarily "steal artists' rice bowls", but can lower the threshold for artistic creation and achieve the vision of "creation by all people". "For example, in music creation, people only need to hum a melody, and AI can compose a complete piece of music based on its emotion, style, etc."
He pointed out that the research intends to surpass today's automatic generative AI and allow human creators to participate more in AI art. "The Empathy AI currently being studied can not only allow people who cannot compose music to compose music but also allow music to show the personality of human authors because they do not generate art purely by keywords, but create with their own emotions."
Guo Yike believes that how to make machines express humanity is an important proposition for AI creation. Photo by Hong Kong Commercial Daily reporter Cui Junliang
In addition to assisting human creation, both professors believe that generative AI can help humans learn how to create. "People can learn the structure of music and how to score through complete AI music. AI technology can also tell you the principles behind AI creation, making self-learning composition or other artistic creation easier." said Johnny Poon.
The two professors added that generative AI can allow different artistic media to blend, such as visualizing music, or conversely turning paintings into music, or even creating dances, allowing authors of different categories to easily carry out cross-media creation without becoming all-round artists.
NO.2 AI Wave is Unstoppable
Last year, a contestant in an art competition at the Colorado State Fair in the United States won the championship in the "Digital Art/Digitally Manipulated Photography" category with a work generated by the AI drawing tool Midjourney, which immediately became a hot topic on the Internet.
Some netizens believe that AI-generated art is not an original work and angrily scolded the contestant involved for cheating. Some digital illustrators also lamented that art is dead and worried that AI threatens their livelihood.
The work "Théâtre D'opéra Spatial" generated by the AI drawing tool Midjourney won the championship in the "Digital Art/Digitally Manipulated Photography" category.
A similar situation occurred again recently in the famous Sony World Photography Awards. The photo that won the Creative category was actually generated by AI images. The winning "photographer" refused to accept the award and scolded the organizer for not being able to distinguish between AI and real-person works.
Facing the many controversies of AI art, Guo Yike retorted that these arguments are meaningless: "Technology will progress, and society must develop. Many jobs that relied on manpower two hundred years ago have now been replaced by technology. Researching how to integrate AI technology into life is important, and it is inevitable."
Johnny Poon also used the past to illustrate the present, believing that AI art will not eliminate traditional art, but will become a tool for creation methods, "Just as the invention of the camera did not replace painting, but also derived new art forms such as photography and film. Good-looking and good-sounding works will not be eliminated by time and technology."
As a musician, Johnny Poon believes that AI can become an auxiliary tool for artistic creation. Photo by Hong Kong Commercial Daily reporter Cui Junliang
Regarding someone using AI works to win awards in art competitions, Guo Yike does not deny that AI art will replace some artists, but he firmly believes that AI will not make all future artists lose their jobs: "Because future artists will improve due to technology."
NO.3 Artificial Intelligence Trains Students' Critical Ability
Generative AI weakening human desire to create is just one of the controversies, followed by the issue of originality.
The AI chatbot software that has been hotly discussed around the world in recent months can be said to be the target of public criticism. This chatbot can not only translate, proofread, and program but also write poems and lyrics, create stories and scripts, and write papers and reports, and its ability to imitate real-person writing is extremely high.
In the academic world that maintains academic integrity and values originality, it has caused great controversy. Schools and teachers are worried that students will abuse AI for ghostwriting and cheating.
Not Mind Students Using AI to Write Papers
Regarding the assistance of generative AI, or even replacing students to do homework, Guo Yike answered with a serious face: "I don't object to students using AI to assist in writing papers, and I even think it is not a problem for them to use AI for exams."
This answer somewhat surprised the reporter, and I believe the students of the school will be extremely excited.
Professor Guo explained that using generative AI to do the work does not mean that students can just sit back and relax: "Learning is not about answering, but about asking. People only start learning when they have questions. Papers, in the final analysis, test whether students can ask the right questions. AI simplifies the paper to its essence, just as the Internet saves students the time of going to the library to look up information. AI also saves students the time of looking up and summarizing information on the Internet, allowing them to focus more on asking more and better questions. If all students use AI, there may be no more failures, but from their ability to ask questions, we can distinguish between high and low."
As the author who won the grand prize using the AI drawing tool mentioned above said, in the process of "creation", he had to constantly adjust keywords to the AI system, and the work underwent more than 900 iterations and took 80 hours to complete.
Generative AI undoubtedly simplifies the process of traditional creation, but it still tests the creator's critical thinking ability. In this regard, Professor Poon added: "AI-generated academic answers may not be completely accurate, which can also train the critical thinking and editing skills of students using AI."
Source: Hong Kong Commercial Daily
Hong Kong Science Popularization and Science Fiction Academy (HKSPSF) promotes scientific knowledge and sci-fi culture, and fosters the popularization and development of science education.