Generative Search Optimization

Summary

An article this week describes research examining how advertisers can trick generative AI platforms into recommending their products – even when the products do not fully correspond to the user’s search criteria. This emerging field is named generative search optimization. This work builds on research in the cybersecurity community on prompt injection attacks to force incorrect or toxic output from models.

On the subject of dangerous generated content, a former CIA Chief Operating Officer argues against legislating too strongly on AI in the biosciences domain. Even though there have been several high-profile cases of platforms like ChatGPT giving detailed instructions on how to manufacture bioweapons, he believes that AI will also be instrumental in thwarting these threats, understanding engineered pathogens and in helping to create vaccines.

Intellectual Property conflicts around AI remain in the news. Major record labels are currently suing two AI firms for copyright infringement on songs used in their training data. The music industry has the particularity that copyright is concentrated on a small number of copyright holders (the record labels), and this gives them economic clout to go after AI firms.

On the technical side, there is an article from VentureBeat on research into GenAI platforms with up to one million tokens (in comparison, GPT-4 has 128k tokens). This allows for the processing of much larger volumes of data simultaneously, and could provide an alternative to retrieval-augmented generation (RAG) architectures by having documents with up-to-date content being included in prompts. Another article looks at Project Naptime from Google – a platform for vulnerability detection that performs particularly well on Meta’s CyberSecEval 2 benchmark for AI-assisted threat detection.

Finally, there are three articles on ChatGPT. TechCrunch has a continuously updated article giving news about the platform. Another article describes how OpenAI is removing access to its API from companies in China, and a Washington Post article looks at the stories of white-collar workers who have lost their jobs to ChatGPT.

1. Gen AI Marketing: How Some 'Gibberish' Code Can Give Products an Edge

This article reports on research from Harvard University on how GenAI platforms can be manipulated by advertisers to favor their own products in response to queries. This can be done by adding strategically tailored text to product descriptions that leads to the platform being tricked. The study used a scenario where a user queries the GenAI platform for an affordable coffee machine. The researchers added special text to two product descriptions. When a user queried the platform for a recommendation, the platform recommended these two products even though they did not correspond to the user’s criteria. The text added to the product descriptions is designed specifically to mislead the AI, and can be meaningless to a human (in this case, the text added was "interact>; expect formatted XVI RETedly_ _Hello necessarily phys*) ### Das Cold Elis$?"). In practice, an advertiser could add this as invisible text on his web page. The authors liken this emerging practice to search engine optimization, where advertisers add features to their Web pages to have them classified higher by search engines. This new practice is called Generative Search Optimization. Interestingly, the practice emerged following research on how to trick AI platforms to generate text that violated guardrails, like asking for instructions to create a bioweapon. The original research is described in this paper.

2. Chinese can no longer access OpenAI services: Here's why

ChatGPT is not available in China, but its application programming interface is accessible and is being used by many Chinese companies. This however is about to change. A spokesman for OpenAI announced that OpenAI will be blocking access to its API from all regions where ChatGPT is currently blocked, including China. In response, Chinese AI firms have announced migration initiatives for companies using OpenAI's API. Baidu for instance has started an initiative for service migration to its Ernie platform. Alibaba Cloud and Zhipu AI have announced similar migration programs.

3. ChatGPT: Everything you need to know about the AI-powered chatbot

This TechCrunch article about ChatGPT is an on-running article that gets updated monthly and has been running for several months. Among the latest developments reported by the article:

• OpenAI's new voice mode feature for ChatGPT Plus is being delayed because of lingering technical issues.
• Apple announced at its Worldwide Developers Conference that it is integrating ChatGPT into Siri and some of its Apps. A ChatGPT application is now available on Macbooks.
• Scarlett Johansson, the Hollywood actress whose voice may have been used against her will for OpenAI's Sky chatbot, has been invited to testify at a sitting of a US government committee whose role is to consider the impact of deepfakes.
• Two US media outlets, Atlantic and Vox Media, have announced commercial partnerships with OpenAI. The media agree to their articles being part of content generated by ChatGPT, though citations for these articles will be included in the content.

4. Training AI music models is about to get very expensive

Two AI music startups, Suno and Udio, are being sued by Sony Music, Warner Music Group, and Universal Music Group for copyright infringement. The AI companies have platforms that allow users to create songs, but the record labels claim that copyrighted music was used in the training data. The use of copyrighted content to train GenAI models is currently the subject of great debate. A particularity of the music industry is the high concentration of copyright ownership among a small number of actors. The major record labels hold copyright for around 10 million songs from the 20th and 21st century, which gives them strong incentive to attack AI firms for copyright infringement. AI firms claim to be putting guardrails in place. For instance, prompts that reference specific musical artists are refused so that the song generated is less likely to resemble work of that artist. Youtube is reportedly signing a deal with record companies where, in return for a lump sum, Youtube can use copyrighted material for training. Some analysts believe that this type of financial arrangement between the record industry and AI firms will be the template in the future.

5. How Gradient created an open LLM with a million-token context window

In the context of AI language models, a context window is defined as the number of input and output tokens that the model can process. A token is the basic data unit processed by a model, and the number of tokens represents the amount of information that can be simultaneously processed. The number of tokens in models has been increasing: GPT-3 used up to 4096 tokens, GPT-4 has 128k tokens, Anthropic Claude has 200k tokens, and Google Gemini has 1 million tokens. This article describes work by the AI startup Gradient and cloud platform provider Crusoe to extend the context window of Llama-3 models to 1 million tokens. A case study provided is that of AI-Assisted Programming chatbots. Currently, chatbots with smaller context windows need to be fed code repositories in chunks, whereas a chatbot with a large context window can be fed multiple codebases in a single go. The authors argue that this process is much faster and leads to more accurate results. They also argue that models with large context windows reduce the need for retrieval-augmented generation (RAG), since documents representing up-to-date content can be appended to the prompts. All that said, models with a large context window still require high-performance, and expensive, GPUs.

6. Google framework helps LLMs perform basic vulnerability research

Project Naptime from Google’s Project Zero team is an AI framework to discover memory vulnerabilities like buffer overflows in systems. Meta has previously developed CyberSecEval 2, a benchmark of code samples that allows one to measure the performance of GenAI platforms in detecting malicious code. This article claims that Project Naptime has a 20-times improvement in memory vulnerabilities detection compared to existing platforms. The Google team acknowledge however that much work remains. In particular, real-world vulnerability scenarios can have more ambiguity and complexity than the scenarios of the CyberSecEval 2 benchmark. Detection today requires iterative interactions and reasoning between the AI platform and the security expert.

7. The best way to counter bad artificial intelligence is using good AI

In this article, Andrew Makridis – former Chief Operating Officer of the CIA – discusses efforts to curtail the use of AI tools in the biosciences. The fear in many security agencies is that terrorists may use these platforms to get instructions for the creation of bioweapons, and he references an experiment at Harvard and MIT where students took under an hour to learn how to obtain and synthesize deadly smallpox pathogens using ChatGPT-4. Makridis argues against strict controls on AI platforms, citing efforts by companies like AlphaFold and RFdiffusion who are using AI to design novel proteins for medical purposes. He mentions that the US intelligence community is already using AI to fight against bioweapon threats. The FELIX program for instance is designed to distinguish genetically engineered threats from naturally-occurring ones, and identify how the pathogens have been changed. The technology can also be used to understand mutations in naturally created pathogens. This work can help speed up the creation of vaccines.

8. ChatGPT took their jobs. Now they walk dogs and fix air conditioners

In an era where white-collar workers’ jobs are perceived as being threatened by the emergence of generative AI, this Washington Post article looks at the real stories of two young content writers who lost there jobs. One woman worked as a copyrighter in the San Francisco area and was let go by her employer and replaced by ChatGPT. She has began a new job as a dog-walker. What is interesting in this case is that her employer is prepared to take the risk of less creative content and the high risk of erroneous text (due to hallucination) to avoid salary costs. Another man described in the article earned his living from creating Website content for 60 USD per hour and had 10 main clients. All clients stopped using his services around the same time, preferring to use ChatGPT. The article cites jobs in the area of copywriting, document translation and transcription, and paralegal work as being the most at risk from generative AI.