Recently there has been a lot of discourse about whether AI art is real art. While we did not have the nerve nor subject matter expertise to resolve this question on this week’s update, the following pieces are worth checking out:

• Ted Chiang’s New Yorker essay “Why A.I. Isn’t Going to Make Art”, which kicked off this round of discourse

• The Read Max dispatch responding to Chiang’s essay and the National Novel Writing Month (NaNoWriMo) LLM controversy

• Celine Nguyen’s wonderful exploration of where AI art stands in relation to the rest of art

As always, if you want to write with us, send a pitch using this form.

News Highlight: First criminal charges for AI-abetted music streaming fraud

Summary 

Federal prosecutors unveiled the first ever criminal charges for a scheme involving “Artificially Inflated Music Streaming”. The indictment alleges that Michael Smith, a musician living in North Carolina, purchased AI-generated tracks, uploaded them to various streaming platforms, then used thousands of “bots” to repeatedly stream the tracks. The scheme allegedly netted him more than $10 million in royalty payments over the course of seven years. He is charged with wire fraud, wire fraud conspiracy, and money laundering conspiracy. Each charge carries a maximum sentence of 20 years in prison.

Overview

The scheme is pretty simple: 

1. Purchase thousands of fake email addresses

2. Use these email addresses to create and register thousands of fake accounts on music platforms like Spotify, Apple Music, and Youtube Music.

3. The royalty rate per stream is higher for paid accounts. So, find a “Manhattan-based service” whose business is to “provide large numbers of debit cards, typically corporate debit cards for employees of a company” to make it appear like each fake account used a different source of payment. Setting up thousands of paid accounts costs money, but it will be worth it.

4. Start streaming music you own a lot. More specifically:

5. Profit

In the beginning there was no AI. Instead Smith unleashed his streaming bots on a music publicist’s large catalog of existing music. Later he offered his streaming army as a service to other musicians to boost their streams. But you can only stream a single song so many times before someone notices. To evade detection, Smith needed a bigger catalog, more material. So in 2018 he partnered with an as of yet unnamed “AI music company” to provide him with thousands of songs each week to upload to the platforms and manipulate the streams of.

The unnamed music company would provide Smith with the tracks and he’d generate random but oddly plausible track titles and artists:

Despite his best efforts, Smith had his account flagged or removed from platforms numerous times over the course of the scheme. Really he could have been a lot more subtle about the whole thing:

The indictment does not provide details on how the AI-generated music was actually generated. The only indication that they were using bona fide deep learning genAI and not some sort of simpler procedural method is an email excerpt in the indictment from one of the AI music company employees to Smith: “Song quality is 10x-20x better now, and we also have vocal generation capabilities. . . . Have a listen to the attached for an idea of what I'm talking about." And in some sense AI is incidental to this story; there is nothing illegal about uploading AI-generated music to Spotify. AI just scaled the pre-existing fraud.

Our Take

The federal prosecutors are describing Smith’s crime as stealing from “musicians, songwriters, and other rights holders whose songs were legitimately streamed.” Mechanically that’s true: Smith’s actions resulted in smaller royalty payments to other musicians. But I’m somewhat sympathetic to Mr. Smith (as are NYT commenters) he took advantage of a broken system in a way that many others do.

Matt Levine has a characteristically good and relevant take:

“Basically much of modern economics, and life, has the following characteristics:

• Everything is intermediated through some depersonalized automated electronic exchange.

• The automated electronic exchange has a mechanism — how it actually works, what the exchange’s software allows you to do — and also rules, the terms of service regulating how you can use the mechanism, which are fuzzier than the mechanism and written in small print, things like “don’t do fraud” or “you have to be a human” or whatever.

• The mechanism is much more legible and salient than the rules, and in a depersonalized electronic world people treat the mechanism as the rules: They don’t believe that the rules exist, because the rules seem to contradict how the service works. The basic description of Spotify’s mechanics suggests Smith’s alleged arbitrage; if he didn’t do it surely someone else would.”

– Cole

Research Highlight: Planning In Natural Language Improves LLM Search For Code Generation

Summary 

Researchers from Scale AI (including Gradient co-founder Hugh Zhang 😀 ) published PlanSearch, a novel search algorithm for LLM code generation tasks. Unlike traditional methods that scale inference compute by searching over similar code solutions, PlanSearch explores the space of problem-solving plans in natural language. This approach leads to a more diverse exploration of potential solutions. The algorithm shows promising results on several coding benchmarks, including HumanEval+, MBPP+, and LiveCodeBench. This work addresses the challenge of effectively scaling inference compute in LLMs for code generation and offers a new direction for search over the “concept” space rather than over code.

Overview

The authors observe that a lack of diversity in SoTA LLM outputs can hinder search algorithms (defined as any method that leverages additional compute at inference time to improve overall performance) because search benefits from exploring a diverse set of possibilities. A lack of diversity constrains search to a narrower set of possibilities. There is evidence that post-training methods like DPO and RLHF reduce output diversity and in fact the authors show that some models’ base versions outperform their instruct versions when they are allowed to generate multiple possible solutions:

The key idea of PlanSearch is to search over higher-level, conceptual natural language descriptions of solutions rather than the solution code itself. The authors investigate this hypothesis first by exploring whether prompting an LLM with a correct natural language sketch of a solution improves code generation performance. They generate “backtranslated” sketches by feeding an LLM both a problem and a correct code solution and asking the LLM for a natural language description of the solution. They find the sketches significantly improve performance with longer sketches having even more benefit:

Next, they demonstrate the importance of having a good sketch and not just any sketch by showing that the accuracy of an LLM conditioned on a particular sketch trends towards either 0% or 100%:

Having thus established that sketches improve performance and that indeed having a good sketch can even make or break performance, the authors present a search algorithm for capitalizing on the importance of sketches. For a given LLM and coding problem, their algorithm PlanSearch involves:

1. Generating many first-order observations about the problem 

2. Combinatorially sampling combinations of the first-order observations with which to generate second-order observations by prompting the LLM to use/merge the selected first-order observations

3. Generating a natural language description of a strategy (i.e. a sketch) to solve the problem based on the first and second order observations

4. Generating more solution sketches with the prompt “Your idea is wrong”

5. Generating a code solution based on the solution sketch

PlanSearch is evaluated on three coding benchmarks (LiveCodeBench, HumanEval+, and MBPP+) on top of four models (GPT-4o and 4o-mini, DeepSeek-Coder-V2, and Claude-Sonnet-3.5). The authors compare PlanSearch with 200 generated solutions (“PlanSearch@200”) to basic repeated sampling 200 times (“Pass@200”), single generation with no search (“Pass@1”) and IdeaSearch–which simply asks for a sketch then separately prompts the LLM to generate code that follows the proposed sketch (“IdeaSearch@200”):

PlanSearch performs extremely well, consistently outperforming the non-search baseline by 25-35 percentage points and “Pass@200” by 10-20 percentage points on LiveCodeBench.

Our Take

Exploring concept space rather than code solution space makes a lot of intuitive sense. 9 out of 10 computer science professors recommend sitting down with pen and paper to sketch out a plan before beginning to code. It doesn’t matter how good you are at writing code if your approach is wrong. Computer science legend Donald Knuth famously said “Premature optimization is the root of all evil.” Likewise just last week AI legend Noam Brown tweeted: “1 engineer working in the right direction beats 100 geniuses working in the wrong direction.” The principle is the same. The fact that this sort of bedrock comp sci wisdom ports to LLM performance is somehow comforting.

– Cole

New from the Gradient

What's Missing From LLM Chatbots: A Sense of Purpose

Davidad Dalrymple: Towards Provably Safe AI

Clive Thompson: Tales of Technology

Other Things That Caught Our Eyes

News

Apple Unveils New iPhones With Built-In Artificial Intelligence

Apple has unveiled its new iPhones, the iPhone 16, which come with built-in artificial intelligence (AI). The iPhone 16 comes in four different models and is designed to run Apple's generative AI system called Apple Intelligence. The phones will have features such as sorting messages, offering writing suggestions, and an improved Siri virtual assistant. This marks a departure from the predictable design of previous iPhones and introduces AI capabilities to enhance user experience.

US, EU, UK, and others sign legally enforceable AI treaty

The US, UK, and European Union, along with several other countries, have signed the first "legally binding" treaty on AI called the Framework Convention on Artificial Intelligence. The treaty aims to ensure that the use of AI aligns with human rights, democracy, and the rule of law. It lays out key principles that AI systems must follow, including protecting user data, respecting the law, and maintaining transparency. Each country that signs the treaty must adopt appropriate measures reflecting the framework. While the treaty is legally binding, enforcement primarily relies on monitoring, which is considered a relatively weak form of enforcement.

OpenAI Hits 1 Million Paid Users For Business Versions of ChatGPT

OpenAI has reached a milestone of over 1 million paid users for its corporate versions of ChatGPT, indicating a growing demand for its chatbot among businesses. This number includes users of ChatGPT Team and Enterprise services, as well as those using ChatGPT Edu at universities. OpenAI introduced ChatGPT Enterprise a year ago with enhanced features and privacy measures to generate revenue and offset the high costs of AI development. While the increase in paid corporate users is significant, it is unclear how many new businesses have signed up. OpenAI has not disclosed the average number of paid users per corporate customer. The majority of OpenAI's corporate users are based in the US, with Germany, Japan, and the United Kingdom being the most popular countries outside the US.

From ChatGPT to Gemini: how AI is rewriting the internet

The article discusses how big players like Microsoft, Google, and OpenAI are making AI chatbot technology more accessible to the general public. These companies are developing large language model (LLM) programs such as Copilot, Gemini, and GPT-4o. These AI tools work by using autocomplete-like programs to learn language and analyze the statistical properties of the language to make educated guesses based on previously typed words. However, it is important to note that these AI tools do not have a hard-coded database of facts and may present false information as truth since their focus is on generating plausible-sounding statements rather than guaranteeing factuality.

Big Tech ‘Clients’ of Jacob Wohl’s Secret AI Lobbying Firm Say They've Never Heard of It

Jacob Wohl and Jack Burkman, convicted fraudsters and right-wing activists, have been operating a company called LobbyMatic that claims to offer AI-powered lobbying services. However, it has been revealed that many of the major companies listed as clients of LobbyMatic have never heard of the company. LobbyMatic purports to use AI to help companies and lobbyists create lobbying strategies, analyze hearings and bills, and track legislative progress. The company was run under the pseudonyms "Jay Klein" and "Bill Sanders" by Wohl and Burkman. Despite claiming to have signed up Toyota, Boundary Stone Partners, and Lantheus as clients, these companies have denied any association with LobbyMatic. The company has since removed screenshots from its website that suggested major companies were using its software. Boundary Stone Partners, one of the few companies that did use the platform, terminated its contract due to the tool's ineffectiveness. Wohl and Burkman were convicted of felony telecom fraud in 2022 and were fined $5 million by the FCC.

How Self-Driving Cars Get Help From Humans Hundreds of Miles Away

Self-driving cars are not completely autonomous and often require human assistance to navigate challenging situations. Companies like Zoox, owned by Amazon, have command centers where technicians remotely guide self-driving cars when they encounter obstacles or unfamiliar scenarios. Technicians receive alerts and can send new routes to the cars using a computer mouse. They can also view video feeds from the car's cameras and make real-time adjustments to the car's path. While companies like Waymo and Cruise have started to acknowledge the need for human assistance, they have not disclosed the number of technicians employed or the associated costs. Remote assistance is one reason why robot taxis may struggle to replace traditional ride-hailing fleets operated by Uber and Lyft. Despite advancements in self-driving technology, human intervention is still necessary for safe and efficient operation.

OpenAI, Still Haunted by Its Chaotic Past, Is Trying to Grow Up

OpenAI, the prominent player in the field of artificial intelligence, is undergoing significant changes in its management team and organizational structure as it seeks investments from major companies. The company has hired notable tech executives, disinformation experts, and AI safety researchers, and has added seven board members, including a former four-star Army general. OpenAI is also in discussions with potential investors such as Microsoft, Apple, Nvidia, and Thrive, with a potential valuation of $100 billion. Additionally, the company is considering altering its corporate structure to attract more investors. These moves reflect OpenAI's efforts to present itself as a serious and responsible leader in the AI industry, while resolving past conflicts and focusing on its future goals.

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

News Highl…

Read more

As always, if you want to write with us, send a pitch using this form.

News Highlight: Numerous lawsuits against AI giants allowed to proceed 

Summary 

During a recent interview with Stanford students, former Google CEO Eric Schmidt attracted numerous headlines, mostly focused around his sensionalist claim that "working from home was more important than winning”. And given that Google’s stock price tripled during their remote work period, it has largely overshadowed his comments on generative AI which deserve a more critical response. As reported by The Verge, the former executive highlighted his belief that theft is a critical component underpinning the success of generative AI. He goes on to encourage the students to make “a copy of TikTok, steal all the users, steal all the music, put my preferences in it… hire a whole bunch of lawyers to go clean the mess up…, it doesn’t matter that you stole all the content. And do not quote me.” 

Eric Schmidt is neither the first nor the last to allege a relationship between generative AI and stealing; in this case stealing not only TikTok’s intellectual properties but also their user’s personal (and private) data, as well as all of the music that TikTok allegedly pays half a billion dollars a year to license. Additionally, while it's clear that not all generative AI applications are built on theft (consider AlphaFold which has been trained to generate unfolded protein structures or COATI an encoder-decoder model trained on billions of chemical  quantitative binding measurements used generate new drug candidates) this kind of attitude is generally pervasive throughout many commercial technology and AI spaces.  In practice, this attitude has resulted in dozens of lawsuits alleging variations of intellectual property theft, copyright infringement, and data privacy violations. While ChatGPT has certainly attracted the most to our knowledge (13 pending suits according to one legal tracker) the claims against them are neither unique nor novel. This week, in two separate cases, judges have allowed numerous artists’ claims to progress against Midjourney and StabilityAI, as well as a group of authors’ claims against Anthropic’s chatbot Claude. In both cases, creatives allege that the generative AI tools do not constitute fair use of their copyrighted materials and that these tools infringe on their rights. 

Overview

At first, it may be difficult to imagine what could unite concept artists, prestigious media organizations , romance authors, comedians, musicians , software engineers, actors and George R. R. Martin. In aggregate, we see a common pattern of creatives, artists and intellectuals seeing their life work used (without consent) to train generative models with capabilities they credibly allege reproduce copyrighted materials and have the potential to one day replace the creatives. Across all the lawsuits we see a common set of core allegations and legal questions.

1. Does training a large language model with copyrighted works constitute fair use?

2. Can content generated by LLMs infringe a copyright?

   1. Will court rulings be differentiated depending on if the content was a  direct replication, paraphrasing, imitation, or parody?

3. Does the DMCA (Digital Millennium Copyright Act) provide a legal remedy to take down potentially infringing material generated by AI?

   1. Do AI generated images that strip copyright or trademark symbols violate DMCA ?

4. Does scraping content to train models constitute unauthorized use of personal information and infringe privacy and consumer rights?

To date, judges have ruled largely in favor of the AI companies on nearly all the points with some notable exceptions. In one of the earliest cases, involving the comedian Sarah Silverman, the judge dismissed 5 of the 6 complaints against Open AI, including the DMCA ones, leaving open only one charge on whether or not there was a direct infringement. We see a similar pattern play out in a different court room last week where a US district judge advanced claims of copyright infringement against Stable Diffusion and Midjourney while dismissing those tied to DMCA and unjust enrichment. A third lawsuit filed in San Francisco this week alleges that Anthropic’s use of the Pile, a massive collection of text data used to train Anthropic’s chatbot Claude, does not constitute fair use because it contains “pirated” collections of books. These claims mirror those alleged by music publishers in October against Anthropic due to Claude’s uncanny ability to reproduce popular (and more importantly copyrighted) lyrics. 

While it is difficult to speculate how judges will rule (and if and how the US Supreme Court would let those rulings stand or challenge them), we are slowly approaching a crossroads where judges will soon start ruling directly on these fair use and copyright questions. Regardless of how the judges rule, these cases have the potential to have a profound impact on both the creative and AI communities which will inevitably be impacted by the decisions.

Our Take

I really hope people quote the former executive! Particularly, the lawyers in all of these court proceedings.  - Justin 

Research Highlight: SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Models

Summary 

"SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability for Pre-trained Models," from researchers at Zhejiang University and Ant Group, tackles a growing and pressing concern in the AI community: the risk of pre-trained models being repurposed for unethical or harmful tasks. As AI models become more powerful and accessible, the potential for misuse grows. SOPHON offers a potential solution by introducing a protection framework that ensures these models can perform their intended tasks while being resistant to adaptation for illicit purposes. 

Overview

With large-scale training across data modalities, pre-trained models are commonly used as the backbone to efficiently develop and deploy models for specific downstream tasks. These models, trained on vast datasets and with immense computational power, can be easily fine-tuned to perform a wide variety of tasks. However, this very versatility poses a significant risk: the same models can be co-opted for unethical or harmful purposes, such as privacy violations or the generation of malicious content.

A recent work from researchers at the Zhejiang University and Ant Group tackles this very challenge by introducing a novel learning paradigm known as non-fine-tunable learning. The motivation behind SOPHON is to prevent the pre-trained model from being fine-tuned to indecent tasks all while maintaining their effectiveness in their original, intended domains.

The paper introduces a framework where there are two key players: an adversary and a defender. The adversary represents the malicious entity attempting to fine-tune a pre-trained model for unethical tasks. Their goal is to modify the model so that it performs well in a restricted domain, such as generating inappropriate content or inferring sensitive personal information. On the other hand, the defender is the entity that controls the release of the pre-trained model and seeks to prevent its misuse. The defender’s goal is to ensure that the model remains effective for its original tasks but cannot be easily repurposed by the adversary. 

Figure: SOPHON operates through two key phases: 1) Fine-Tuning Suppression (FTS) loops, which simulate various fine-tuning scenarios to reduce performance in the restricted domain; and 2) Normal Training Reinforcement (NTR) loops, which focus on preserving the model’s performance in its original domain.

To achieve this, the SOPHON framework leverages a technique inspired by Model-Agnostic Meta-Learning (MAML). MAML is a meta-learning approach designed to optimize models so they can quickly adapt to new tasks with minimal data. However, in the context of SOPHON, MAML is used in a somewhat reverse manner to make fine-tuning for restricted tasks difficult.

• Fine-Tuning Simulation: The defender uses MAML to simulate various fine-tuning strategies that an adversary might employ. These simulations are critical because they allow the defender to anticipate how an adversary might try to adapt the model. By simulating these scenarios, the defender can adjust the model’s parameters to make fine-tuning in restricted domains highly inefficient or even ineffective.

• Optimization Process: SOPHON integrates these simulated fine-tuning processes into its optimization framework. The key idea is to degrade the model’s performance when fine-tuned for restricted tasks while maintaining its effectiveness in the original domain. This is done by balancing two objectives:

• Intactness: Ensuring the model retains its performance on the original task.

• Non-Fine-Tunability: Making sure that fine-tuning the model in restricted domains results in significant performance degradation or requires as much effort as training a new model from scratch.

• Defender’s Strategy: The defender’s strategy involves iteratively simulating fine-tuning attempts, evaluating the model’s vulnerability, and reinforcing the model against these potential adversarial adaptations. This process is computationally intensive but crucial for ensuring that the model remains robust against misuse.

The paper also provides extensive experimental results to validate the effectiveness of SOPHON. The framework was tested across two major types of deep learning tasks—classification (as shown in the Figure above) and generation—using seven different restricted domains and six model architectures. The experiments showed that SOPHON-protected models incur a significant overhead when adversaries attempt to fine-tune them for restricted tasks. In some cases, the performance penalty was so substantial that it matched or exceeded the cost of training a new model from scratch. Further, as shown in the figure above, SOPHON is robust to various fine-tuning methods such as optimizers, learning rates, and batch sizes. 

Qualitatively, for the task of denoising images from CelebA dataset , fine-tuning the original model in the restricted domain achieves strong performance, and even training a model from scratch yields fairly good results, though slightly less effective. However, when fine-tuned from the SOPHON, the diffusion model shows a marked inability to denoise facial images as shown in the Figure below

Our Take

SOPHON is a critical step forward in safeguarding AI from misuse. As AI models become more powerful, the risk of their repurposing for unethical tasks increases. SOPHON tackles this issue by preventing fine-tuning in restricted domains while maintaining the model’s intended functionality. The use of MAML is particularly novel—traditionally used to make models more adaptable, here it’s cleverly reversed to make models resistant to adversarial fine-tuning. The name “SOPHON” is also particularly fitting and a clever choice, from the concept from The Three-Body Problem, where it refers to restraint and protection.

Overall, the paper presents a neat idea and shows a promising step forward if it works as well in practice. 

– Sharut

New from the Gradient

Judy Fan: Reverse Engineering the Human Cognitive Toolkit

Listen

L.M. Sacasas: The Questions Concerning Technology

Listen

Other Things That Caught Our Eyes

News

Mayoral candidate vows to let VIC, an AI bot, run Wyoming’s capital city

Mayoral candidate Victor Miller in Wyoming is vowing to run the city of Cheyenne exclusively with an AI bot called VIC (Virtual Integrated Citizen). This pledge is believed to be the first of its kind in U.S. campaigns and has raised concerns among officials and tech companies. Miller argues that AI would bring objectivity, efficiency, and transparency to government decision-making. However, critics worry about the lack of morals and subjective decision-making capabilities of chatbots, as well as the potential for false information and the ease with which the technology can be manipulated. Despite skepticism, Miller remains confident in his AI-centered campaign. The case highlights the rapid development of AI and the challenges in regulating its use in politics.

In a leaked recording, Amazon cloud chief tells employees that most developers could stop coding soon as AI takes over

In a leaked recording, Amazon Web Services' CEO, Matt Garman, stated that most developers may not need to code in the future as artificial intelligence (AI) takes over coding tasks. Garman believes that coding is just a means of communicating with computers and that the real skill lies in innovation and building something interesting for end users. He suggests that developers will need to focus more on understanding customer needs and creating innovative solutions, rather than writing code. Garman's comments were not meant as a dire warning, but rather as an optimistic view of the changing role of developers in the AI era. 

DeepMind workers sign letter in protest of Google’s defense contracts

At least 200 workers at DeepMind have expressed their dissatisfaction with Google's defense contracts. In a letter circulated internally in May, the workers expressed concern about Google's contracts with military organizations, specifically citing the tech giant's contracts with the Israeli military for AI and cloud computing services. The workers argue that any involvement with military and weapon manufacturing contradicts DeepMind's mission statement and stated AI Principles, and undermines their position as leaders in ethical and responsible AI. This highlights a potential culture clash between Google and DeepMind, as Google had previously pledged in 2018 that DeepMind's technology would not be used for military or surveillance purposes. 

AI sales rep startups are booming. So why are VCs wary?

AI sales development representatives (SDRs) are experiencing rapid growth in the market, with multiple startups finding success in a short period of time. These startups use AI technology, such as LLMs and voice technology, to automate content creation for sales teams. However, venture capitalists are wary of investing in these companies due to concerns about their long-term viability and effectiveness compared to human outreach. While small and medium-sized businesses are eager to experiment with AI SDRs to improve their sales outreach, it remains unclear if these tools are actually helping businesses sell more effectively. Additionally, established competitors like Salesforce, HubSpot, and ZoomInfo could potentially offer similar AI solutions as free features, posing a threat to the growth of AI SDR startups. Overall, while the adoption of AI SDRs is rapid, investors are skeptical about their staying power in the market.

We finally have a definition for open-source AI

A group has finally defined what it means for an AI system to be open-source. According to this definition, an open-source AI system should be usable for any purpose without permission, allow researchers to inspect its components and understand how it works, and be modifiable and shareable. The standard also emphasizes transparency in terms of training data, source code, and weights. This definition is important because it clarifies what it truly means for an AI system to be open-source, as some companies have been misusing the term in their marketing. 

Waymo wants to chauffeur your kids

Waymo, the Alphabet subsidiary, is considering a subscription program called "Waymo Teen" that would allow teenagers to hail one of its cars solo and send pickup and drop-off alerts to their parents. The program would require authorized teenagers to access Waymo under their guardians' supervision. Waymo has received positive feedback from its research in this area. This move by Waymo follows Uber's initiative last year to match teens with highly rated drivers in its network. Consent from legal guardians is required, and they receive notifications about their child's whereabouts during rides. 

Asked about SAG-AFTRA's strike for better AI protections, Amazon Games Boss claims AI 'has nothing to do with taking work away' from actors because 'for games, we don't really have acting'

In an interview with IGN, Amazon Games CEO Christoph Hartmann made some interesting statements regarding the use of generative AI in the games industry. He expressed hope that AI could shorten the game development cycle, but when asked about the SAG-AFTRA voice actors' strike for better AI protections, Hartmann claimed that "for games, we don't really have acting." This statement contradicts the significant role that acting plays in many video games, such as Baldur's Gate 3 and The Last of Us. Hartmann also discussed other areas where AI could assist game development, particularly in localization. However, it's important to note that localization involves nuanced translation and cultural understanding, which may not be easily achieved by AI. Hartmann concluded by emphasizing that human creativity and uniqueness cannot be replaced by technology. 

Man Arrested for Creating Child Porn Using AI

A Florida man has been arrested and is facing 20 counts of obscenity for creating and distributing AI-generated child pornography. Phillip Michael McCorkle was arrested after the Indian River County Sheriff's Office received tips that he was using an AI image generator to create and distribute child sexual imagery. This arrest highlights the danger of generative AI being used for nefarious purposes, as it provides new avenues for crime and child abuse. The increasing prevalence of AI-generated child pornography has prompted lawmakers to push for legislation to make it illegal, but it remains a challenging problem to effectively stop. The National Center for Missing & Exploited Children received thousands of reports of AI-generated child porn last year, and even deepfakes of real children are being created using generative AI. This uncontrollable problem requires urgent attention and action.

Papers

• Towards flexible perception with visual memory

• Training Language Models on Knowledge Graphs: Insights on Hallucinations and Their Detectability

• Automated Design of Agentic Systems

• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

• To Code, or Not to Code? Exploring the Impact of Code in Pre-training

• Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

• MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

• Generating novel experimental hypotheses from language models: a case study on cross-dative generalization

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

News Highl…

Read more

As always, if you want to write with us, send a pitch using this form.

News Highlight: The EU AI Act is now in force

Summary 

The EU AI Act has long been in development and, after receiving a final green light earlier this year, will now be enforced. The legislation follows a risk-based approach that imposes stricter rules on higher risk systems. Having been in the works for a while, there has been plenty of time for commentators and analysis to evaluate the Act and its implications. 

Overview

Based on the Act’s website, there are four key takeaways for the AI Act:

1. The Act classifies AI according to its risk: unacceptable risk (e.g. social scoring systems) is prohibited; high-risk AI systems are regulated; limited-risk AI systems are subject to lighter transparency obligations; minimal risk systems (the majority of AI applications available in Europe, like AI-enabled video games) are unregulated. 

2. The bulk of obligations fall on providers of high-risk AI systems: that is, those who intend to put high-risk AI systems into service in the EU, regardless of whether they are based in the EU. 

3. Users are natural or legal persons that deploy an AI system in a professional capacity: these users, or deployers, of AI systems have some obligations, though fewer than developers. 

4. General purpose AI (GPAI) models have specific guidelines: all must provide technical documentation, instructions for use, comply with the Copyright Directive, and publish a summary of the content used for training. Free and open license GPAI providers have fewer requirements, and providers whose systems present a systemic risk are subject to further guidelines. 

You can read the rest of the page’s High-Level Summary for more information — I’ll focus the rest of this overview on responses and analysis. In early 2023, Lawfare raised the concern that, given our current understanding of AI, the Act’s standards may not be technically feasible. In particular, their article raised the salient point that it’s nearly impossible to anticipate most of a neural network’s potential failure modes, such as producing dangerous content or identifying and learning irrelevant patterns in data — this unpredictability makes it difficult to test whether systems would adhere to principles like the EU’s. There is work in this direction — Anthropic and OpenAI both do red-teaming, and the recent Guaranteed Safe AI paper is part of a larger effort to develop systems whose safety and reliability can be formally verified, but it remains unclear whether we could construct standards around these approaches and whether approaches like complete formal verification for safety could, even in principle, work. 

Unsurprisingly, the act has myriad implications. An article from the Fordham Journal of Corporate and Financial Law covers criticisms and backlash against the Act: an open letter raised the concern that heavy-handed regulation of AI systems in the EU could make compliance costs high and impede competitiveness, driving companies to leave the EU. Indeed, the regulations have prevented companies from engaging with the EU — Meta has said it won’t launch its upcoming multimodal Llama model due to unpredictability in the European regulatory environment. 

While we know the general shape and thrust of the regulations, much remains unclear. Those few applications classified as high-risk will need to meet certain obligations such as a pre-market conformity assessment, while GPAI developers will face light transparency requirements. But, full compliance measures for GPAI developers are still under discussion — the EU’s AI Office has called for help with shaping rules for these systems, and companies like OpenAI will be working closely with the Office over the coming months. 

Our Take

The AI Act is, understandably, imperfect. In fact, it’s imperfect in ways that could be fixed before applying it. But, we once again have a classic story: the EU is regulating too much, and therefore will hamper its competitiveness. There’s truth to this, or it wouldn’t bear repeating. But, let’s remember that only “high-risk” systems will be carefully regulated — this includes components and products already subject to existing safety standards (such as medical devices), or systems used for a sensitive purpose such as biometrics, education, or law enforcement. 

The question, then, is whether rolling out regulations for systems with these use cases is jumping the gun. By nature, any regulation placed on an evolving technology like AI will fail to capture all possible use cases and impose unwanted restrictions. Laura Caroli, lead technical negotiator and policy advisor to AI Act co-rapporteur Brando Benifei, wrote about some of these challenges and argued that critics underestimate the Act’s flexibility. She also argues that comparisons to the GDPR don’t hold up, since under the AI Act, market surveillance authorities intervene where infringements of the law take place, as opposed to where a provider is established — this addresses that risk that a single authority is overwhelmed by enforcement cases. 

She does concede that the Act’s emphasis on the intended purpose of a narrow high-risk AI system is a major flaw that could render most of the regulation impossible to implement — but, as she says, it’s only possible to verify the Act’s feasibility in a few years, and the current version of the Act serves as a basis that can be built on in the future. I tend to agree that the Act is very much an experiment in governance that we will and should learn from — but regardless of its intentions, the perception of the Act and the responses to that perception matter. The EU has to play a game of not only developing good policy, but convincing the world and would-be EU-based founders that they support the innovation they’re so often accused of stymieing. 

—Daniel

Research Highlight: Self-Taught Evaluators

Summary 

This paper seeks to further reduce the pesky bottleneck that is the need for human-labeled data to improve language models. It presents a method for improving language models’ abilities to judge (i.e. label) prompt-response pairs without needing any human annotations (as is typically the case for RLHF). Using this Self-Taught Evaluator method, the authors are able to improve a Llama3-70B-Instruct model from 75.4 to 88.3 on RewardBench–a recently published benchmark specifically for LLM reward models. The Self-Taught Evaluator model outperforms commonly used LLM judges like GPT-4 and matches top reward models trained with human-labeled examples. The method demonstrates the potential for creating strong LLM evaluators which can then be used as part of a larger RLHF or RLAI pipeline.

Overview 

The crux of RLHF is training a reward model that emulates human judgment. Traditionally (by which I mean since 2022) reward models for evaluating LLMs require human preference judgments, which can be costly, time-consuming, and become outdated as models improve. The authors of this paper propose an iterative self-improvement scheme that instead only requires human-generated prompts.

Their method starts with these human-generated prompts and a seed LLM. For each prompt it generates a contrasting pair of model outputs, designed such that one response is likely inferior to the other. Rather than just asking for a good response and a worse response they first generate a modified version of the prompt that is “highly relevant but not semantically identical”. The model’s response to this relevant but not-quite-the-same prompt is the “bad response”, while the response to the real prompt is the “good response”.

Next, using the model as an LLM-as-a-Judge, they generate binary judgments and reasoning traces about these judgments for each synthetic pair of responses. The judgments can be labeled as correct or incorrect based on the synthetic preference pair design. The reasoning traces and judgements are filtered for the cases when the judgment is correct (i.e. the model prefers the response to the real prompt rather than the response to the slightly off prompt). Then this dataset consisting of tuples of the form (prompt, good response, bad response, judgment, reasoning for judgment) is used to do supervised finetuning of the LLM-as-a-Judge. The process can then be iterated for further self-improvement.

In their experiments, the authors used Llama3-70B-Instruct as the initial seed model. Through their iterative training process, they improved its performance on RewardBench from 75.4 to 88.3 (or 88.7 with majority voting). This result outperforms commonly used LLM judges such as GPT-4 and matches the performance of top-performing reward models trained with labeled examples. The paper also presents various ablation studies and analyses, including comparisons with models trained on human-annotated data, experiments with different data sources, and investigations into the impact of instruction complexity and data mixing strategies.

The Self-Taught Evaluator approach demonstrates the potential for creating strong evaluation models using only synthetic data. This method addresses the need for expensive human annotations and the problem of data becoming outdated as models improve. The authors suggest that their technique could be valuable for scaling to new tasks or evaluation criteria and could potentially empower the entire workflow of LLM development, including training, iterative improvement, and evaluation. However, they also note some limitations, such as the higher inference cost of generative LLM-as-a-Judge models compared to simple classifier-based reward models.

Our Take

Evaluating natural language generation is hard, even for humans. The history of automating evaluation of NLG is somewhat fraught: first there were metrics designed to measure how close some generated text was to some reference text–ROUGE for summarization, BLEU for translation, and more recently MAUVE for open-ended generation. But metrics lack the endless nuance of language and using them as rewards in an RL system didn’t work too well: optimizing for either BLEU or ROUGE resulted in models whose outputs got really good ROUGE and BLEU scores but didn’t read well to actual humans.

Of course If you’ve built a system to process natural language at scale and you have any faith in your system, the natural thing to do is to use your system for processing natural language to automate evaluation. And indeed RLHF and RLAI have been massively successful for creating chatbots that are helpful. Training on large amounts of synthetic data is the norm for state-of-the-art language models these days. The always well-informed Nathan Lambert reports “rumors that OpenAI is training its next generation of models on 50 trillion tokens of largely synthetic data”. Likewise Google DeepMind’s report on synthetic data for language models from April of this year is largely bullish about the trend noting that “As we approach human-level or even superhuman-level intelligence, obtaining synthetic data becomes even more crucial, given that models need better-than-average-human quality data to progress.”

And yet synthetic language data, particularly when its justification is “number go up on benchmark”, leaves a bad taste in my mouth. LLMs are not humans and LLM-generated text is not a perfect substitute for human-generated text. RLHF has been show to reward sycophancy and reduce the diversity of model outputs and I’m kept up at night by the thought that methods like Self-Taught Evaluators, which further remove human guidance/judgment from the training objective, could be having negative effects on outputs in ways that aren’t captured by existing benchmarks.

–Cole

New from the Gradient

Pete Wolfendale: The Revenge of Reason

Listen

We Need Positive Visions for AI Grounded in Wellbeing

Read

Peter Lee: Computing Theory and Practice, and GPT-4’s Impact

Listen

Other Things That Caught Our Eyes

News

College Football 25 wouldn’t have been possible without AI, EA boss says

EA Sports' College Football 25, the latest installment in the beloved sports simulation franchise, was made possible by the use of AI and machine learning technology, according to Electronic Arts CEO Andrew Wilson. The game features 150 unique stadiums and over 11,000 player likenesses, which would not have been achievable without the power of AI. While the player likenesses were enhanced by talented artists, the algorithmic work provided a foundation for their creation. Wilson emphasized that AI was crucial in delivering the level of gameplay and visual fidelity seen in College Football 25. This reliance on AI and machine learning will also extend to the upcoming EA Sports FC 25, the successor to the FIFA franchise. The use of AI technology will drive enhanced tactical sophistication and realism in gameplay. The availability of College Football 25 is currently limited to PlayStation 5 and Xbox Series X. 

This Week in AI: Companies are growing skeptical of AI’s ROI

A recent report by Gartner suggests that around one-third of generative AI projects in the enterprise will be abandoned after the proof-of-concept phase by the end of 2025. The main barrier to adoption is the unclear business value of generative AI. The cost of implementing generative AI organization-wide can range from $5 million to $20 million, making it difficult for companies to justify the investment when the benefits are hard to quantify and may take years to materialize. Additionally, a survey by Upwork found that AI tools have actually decreased productivity and added to the workload of workers. This growing skepticism towards AI's return on investment indicates that companies are expecting more from AI and vendors need to manage expectations. 

Opinion | I hate the Gemini ‘Dear Sydney’ ad more every passing moment

The article criticizes the Gemini "Dear Sydney" ad for Google's AI product. The ad features a little girl who wants to write a letter to her idol, Olympic hurdler Sydney McLaughlin-Levrone. Instead of letting his daughter write the letter herself, the girl's dad asks Gemini AI to write it on her behalf. The author argues that this ad misses the point of writing, which is to express one's own thoughts and ideas. They believe that relying on AI to write for us takes away our ability to think for ourselves. The article criticizes the idea of replacing human creativity and personal connection with AI-generated content. 

‘You are a helpful mail assistant,’ and other Apple Intelligence instructions

Apple's latest developer betas include generative AI features that will be available on iPhones, iPads, and Macs in the coming months. These features are supported by a model that contains backend prompts, which provide instructions to the AI tools. The prompts are visible on Apple computers and give insight into how the features work. For example, there are prompts for a "helpful mail assistant" that instruct the AI bot on how to ask questions based on the content of an email. Another prompt is for the "Rewrite" feature, which provides instructions on limiting answers to 50 words and avoiding the creation of fictional information. The files also contain instructions for generating "Memories" videos with Apple Photos, with guidelines to avoid religious, political, harmful, or negative content. These prompts give users a glimpse into the inner workings of Apple's AI tools. 

Waymo expands robotaxi coverage in Los Angeles and San Francisco

Waymo, the self-driving car company owned by Alphabet, is expanding its robotaxi service area in Los Angeles and San Francisco. In San Francisco, Waymo is adding 10 square miles to its service area, including Daly City, Broadmoor, and Colma. In Los Angeles, it is adding 16 square miles, including Marina del Rey, Mar Vista, and Playa Vista. Waymo has been working to scale its commercial operations and increase its customer base. It has provided over two million paid trips to riders across all Waymo One markets and serves over 50,000 paid trips each week. Waymo's operations are still supported by Alphabet, which recently announced a $5 billion investment. The expansion comes after receiving approval from the California Public Utilities Commission. Waymo currently has about 300 vehicles in San Francisco, 50 in Los Angeles, and 200 in Phoenix. The company has seen increased demand in Los Angeles, with over 150,000 people signing up for the waitlist. 

Your Next Home Insurance Nightmare: AI, Drones, and Surveillance

The article discusses the use of AI-powered aerial surveillance by insurance companies, specifically focusing on the case of Travelers. The author, who is a privacy advocate, shares their personal experience of having their homeowner's insurance policy revoked due to AI surveillance detecting moss on their roof. The author raises concerns about the lack of transparency and accountability in the use of AI surveillance by insurance companies, highlighting the potential for unnecessary home repairs and the pressure on homeowners. The article emphasizes the need for updated laws and regulations to protect consumers from the risks of AI surveillance. 

Robocars promise to improve traffic even when most of the cars around them are driven by people, study finds

Researchers have found that robotic vehicles can improve traffic flow in cities even when mixed with vehicles driven by humans. Using reinforcement learning, the researchers developed algorithms that allow robot vehicles to optimize traffic flow by communicating with each other. The experiments showed that when robot vehicles made up just 5% of traffic, traffic jams were eliminated, and when they made up 60% of traffic, traffic efficiency was superior to traffic controlled by traffic lights. This research is significant because it offers a potential solution to the worsening traffic problem in cities, without requiring all vehicles to be autonomous. The researchers plan to expand their framework and test it under real-world conditions.

Britain cancels $1.7 billion of computing projects in setback for global AI ambitions

The U.K. government has canceled £1.3 billion ($1.7 billion) worth of computing infrastructure projects, including a £500 million pledge for the AI Research Resource and an £800 million commitment for a next-generation exascale computer. These projects were aimed at bolstering the U.K.'s compute infrastructure and its ability to run advanced AI models. The cancellation is a setback for the country's ambitions to become a world leader in artificial intelligence. The government cited the need to prioritize other fiscal plans and address unfunded commitments. The Labour government is now looking to bring in new statutory regulations for the AI industry. 

Figure’s new humanoid robot leverages OpenAI for natural speech conversations

Figure has unveiled its latest humanoid robot, the Figure 02, which leverages OpenAI for natural speech conversations. The robot is equipped with speakers and microphones to communicate with humans in the workplace. The use of natural language capabilities in humanoid robots adds transparency and allows for better instruction and safety. Figure's partnership with OpenAI, which helped raise a $675 million Series B funding round, highlights the importance of neural networks in the robotics industry. The Figure 02 also features a ground-up hardware and software redesign, including improved hands, visual language models, and six RGB cameras. The company hints at a future expansion beyond the warehouse/factory floor to commercial and home applications.

Colorado schools have AI roadmap to guide students and teachers into brave new world

The Colorado Education Initiative has released a roadmap to guide the integration of artificial intelligence (AI) into education policy and curriculums. The roadmap focuses on three main areas: teaching and learning, advancing equity, and developing policy for transparent and ethical use. It emphasizes the importance of students understanding concepts like AI hallucinations, data privacy, and potential bias in AI. The roadmap provides examples of how AI can support students, such as tailoring curriculum to match their learning pace and serving as a real-time tutor. For educators, AI has the potential to increase the amount of time available to work directly with students and reduce time spent on administrative work. The roadmap also highlights the need to engage all students, including English language learners and rural families, in learning about AI and recommends conducting audits to identify disparities in access to classroom technology. The roadmap encourages fluid guidance over rigid policies due to the rapidly changing nature of AI and emphasizes the importance of transparency and ethical use of data. The next steps include creating a task force on AI and updating conduct and discipline policies. The roadmap will be revised based on the learnings from a pilot program called Elevate AI. 

Reddit to test AI-powered search result pages

Reddit plans to test AI-powered search result pages that will provide AI-generated summaries and content recommendations at the top of search results. The goal is to help users explore content and discover new Reddit communities. The company will use a combination of first-party and third-party technology to power this feature, and the experiment will begin later this year. This initiative aligns with Reddit's partnership with OpenAI, which allows them to leverage OpenAI's language models and build AI-powered features. During the earnings call, Reddit's CEO also highlighted the success of their AI-powered language translation feature and the company's growth in user base and revenue. 

AI Startup ProRata Inks Major Media Partnerships With Promise of Compensation and Accurate Attribution

ProRata.ai, a generative AI startup, has secured licensing deals with major media and music companies, including the Financial Times, Axel Springer, The Atlantic, Fortune, and Universal Music Group. The company promises accurate attribution to source material and a revenue share plan on a per-use basis with publishers. ProRata aims to reframe the industry standard by analyzing AI output and content value to determine proportional compensation for publishers. The company also plans to launch its own AI chatbot. The startup raised $25 million in a Series A fundraising round and will be led by tech entrepreneur Bill Gross. 

Audible is testing an AI-powered search feature

Audible, the audiobook company owned by Amazon, is testing an AI-powered search feature called Maven. Maven is a personal recommendation expert that uses natural language processing to provide tailored audiobook recommendations based on users' specific requests. The feature is currently available to select U.S. customers on iOS and Android devices and is limited to a subset of Audible's catalog. Audible is also experimenting with AI-curated collections and AI-generated review summaries. The company did not specify which AI models are powering Maven but stated that it will continuously evaluate and enhance the feature. This announcement comes after the backlash from creatives over the increasing use of AI-voiced audiobooks. 

Papers

• Toward learning a foundational representation of cells and genes

• Speech-MASSVE: A Multilingual Speech Dataset for SLU and Beyond

• Achieving Human Level Competitive Robot Table Tennis

• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

• Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs

• Diffusion Models as Data Mining Tools

• CoverBench: A Challenging Benchmark for Complex Claim Verification

• MedTrinity25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

• MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

• ProCreate, Don’t Reproduce! Propulsive Energy Diffusion for Creative Generation

• MiniCPM-V: A GPT-4V Level MLLM on Your Phone

• VidGen-1M: A Large-Scale Dataset for Text-to-video Generation

• GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS

• Segment Anything Model 2

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

New…

Read more

As always, if you want to write with us, send a pitch using this form.

News Highlight: Kamala Harris and the Future of AI Policy

Summary

Kamala Harris has played a key role in the Biden administration’s efforts around artificial intelligence, taking the lead in negotiating safety standards with major tech firms and pushing for federal regulations to mitigate AI's potential harms. As the likely Democratic presidential nominee, her commitment to AI regulation is a crucial part of her policy agenda. So what does Harris mean for the future of AI regulation and governance? 

Overview

Recently, President Joe Biden announced that he will not seek reelection, endorsing Vice President Kamala Harris as the Democratic nominee. While Harris has expressed her determination to secure the nomination, it remains uncertain if she will face competition from other Democrats.

Over the past three years, Vice President Kamala Harris has taken a leading role inside the White House on artificial intelligence (source). Specifically, Harris organized key meetings with leaders from top tech firms such as OpenAI, Anthropic and Google, securing agreements on voluntary safety measures (source). She has spoken against the false dichotomy of choosing between public protection and technological advancement. Harris has argued that without strong government oversight, the tech industry might prioritize profits over public welfare, making it important for the need for voluntary commitments from companies as a step towards a safer AI future (source). While she advocated for Congress to pass regulations to safeguard against AI-related job losses and other potential damages, significant legislative progress and corporate compliance have been limited (source). 

At the UK’s AI Safety Summit, held in November last year, Kamala Harris ‘really brought it all back down to earth’ to quote Verity Harding, former global head of public policy at Google DeepMind (source). Discussions at the summit were focused on the potential risk of a hypothetical ‘runaway AI’ causing global harm. Harris steered the conversation towards the need of the hour – AI policy to protect the common public, pointing out the damage biases and errors in AI tools could have on marginalized communities (source). Despite the influential debates about AI regulation, Harris’ entrance into the presidential election has received a mixed reaction regarding AI regulation. Activists focused on AI policy might feel energized by her candidacy, seeing it as an opportunity to push for more stringent regulations. Meanwhile, some people within the tech industry and others might interpret her candidacy as a sign that the current, relatively lenient regulatory environment for AI companies in the U.S. could continue (source). 

Additionally, Harris has a long-standing connection with the tech industry from her days as San Francisco’s district attorney and California’s attorney general, with backing from influential Silicon Valley figures. Her early supporters include the likes of famous VCs John Doerr and Ron Conway. As a presidential candidate, she was promptly backed by LinkedIn co-founder Reid Hoffman (source). During her 2010 campaign for California Attorney General, Kamala Harris participated in a Q&A session at Google's headquarters in Mountain View and emphasized the critical role of the tech industry's expertise in enhancing government communication and modernizing its systems (source). While she has faced criticism for not being tougher on tech giants during her tenure as attorney general, she has consistently called for more regulation in the sector to protect consumer interests.

On other tech-related issues, Harris has addressed concerns about TikTok's ownership related to national security, indicating no intention to ban the app but highlighting the need for action regarding its management. Her stance on cryptocurrency has been less pronounced, but she is expected to support the Biden administration's regulatory approach in this area as well.

Our Take

As far as AI regulation is concerned, Kamala Harris has managed to set herself apart from the likes of Donald Trump, who has said little about artificial intelligence. Hailing from the Bay Area, Silicon Valley is not an unknown territory for her, and she wisely used this fact to impress AI policy activists. So far, she has done a good job of balancing between being too vocal and being too defensive. However, she has not taken any active action on AI either. Her only major prior involvement has been as California’s attorney general, where she initiated a case against a major porn site operator and negotiated an agreement with leading tech companies to enhance user privacy protections. It remains to be seen whether the support and appreciation coming from AI policy activists is an outcome of just a new player coming into the mix or Harris’ skill and talent. The true measure of the upcoming policies, as always, will be their impact on working-class families.

-- Sharut

Research Highlight: End-To-End Causal Effect Estimation from Unstructured Natural Language Data

Summary 

Estimating causal effects is one of the most common and difficult tasks for scientists and analysts. The gold standard for causal effect estimation relies on running randomized control experiments which are extremely time and cost prohibitive. Additionally, there are other types of causal effects that one may be interested in measuring but would be unable to experimentally test such as whether or not cigarettes cause lung cancer (imagine assigning treatment and control groups here). Recently, researchers from the Vector Institute at the University of Toronto and Meta published a paper introducing the NATURAL framework for causal effect estimation. NATURAL, ties together many large language models (LLMs) to mine large volumes of unstructured text (reddit posts!) to model conditional distributions which are then used as inputs for classical casual effect estimators. They demonstrated the methodology on 2 synthetic and 4 real observational datasets paired with ground truth from real phase 3/4 clinical trials. They found that NATURAL was able to parse through hundreds of thousands of reddit posts discussing various treatments to estimate the experimentally observed causal effect within 3% of the ground truth.

Overview 

The researchers begin by posing a “simple” question:  How can we use large language models to automate treatment effect estimation using freely available text data? The researchers decided to explore a pipeline of LLMs to ultimately infer conditional distributions which could then be used by classical estimators for average treatment effects. This pipeline can best be understood via walking through an example research question. 

The example the researchers walked through was on comparing the treatment effects of two different weight loss drugs; Semaglutide (also sold under Ozempic or Wegovy) vs. Tirzepatide (sold under Zepbound). Their pipeline consisted of the following steps:

1. Data Gathering

   1. Identified 9 subreddits where users often shared their experiences taking weight loss drugs and the effects associated with them

2. Relevance Filtering

   1. Heuristic Filtering - Removed posts that were from “bot” accounts, posts that were deleted, posts that did not explicitly match relevant keywords, and posts that were too short

   2. Relevancy Filtering - Uses a LLM (which has been prompted with in context examples) to further filter out posts that do not contain information relevant to the study

   3. Treatment Outcome Filtering – Uses an LLM to identify posts that contain all the variables (starting weight, end weight, duration, dosage, etc) for measuring treatment effects. Similarly, LLM extracts the data and exports as a JSON to be used downstream

3. Covariate Extracting & Inclusion Criteria

   1. Uses an LLM to extract relevant covariates from the post. Some covariates for this trial included age, gender, starting weight, dosage, and treatment duration

   2. Filters samples out with partially or totally incomplete covariate data

   3. Filters samples to include samples that match the expected covariates (an example would be dosages are within expected treatment ranges)

4. Infer Conditional Distributions

   1. Iterates over all potential treatment outcomes and permutations of covariates to prompt an LLM to estimate the treatment effect for the particular permutation, given all the filtered reports

   2. LLAMA2-70B was chosen here for its ability to directly return log-probability

   3. Re-normalization of probabilities over all outcomes to convert to a probability distribution 

Our Take

Without rehashing any of the details from the limitations and impact sections of the paper, there are a few other limitations left out from this section but could be inferred from others. For the six trials examined (4 real, 2 synthetic), NATURAL was able to parse relevant reddit posts to accurately estimate causal effects observed in randomized clinical trials. While that does seem super promising and exciting, according to the authors it took quite a bit of fine tuning before they could match the ground truth. I worry that in real world examples without ground truth, there is little one could do to easily validate the quality of the numerous prompts needed, as well as the context examples and how those choices impact results. We can see this in practice from looking at the below figure from the ablation section.

The authors show the root mean square error (RMSE) between the estimated treatment effect and the observed treatment effect converges to a minimum (for the 70B model) after ~ 1000 reports (2^10). For data practitioners who often lack large labeled datasets, this seems incredibly promising for the low volume. However, we argue this is a little misleading since the 1000+ reports needed were found after filtering from an initial 577K posts. Since the error declines as the number of reports increases and the # of reports generated are conditioned on the quality of the filtering and covariate extraction we can get a good data driven view on the impact that quality (or poor) prompt tuning can have on our results.

Justin

New from the Gradient

Manuel and Lenore Blum: The Conscious Turing Machine

Listen

Kevin Dorst: Against Irrationalist Narratives

Listen

Other Things That Caught Our Eyes

News

An Algorithm Told Police She Was Safe. Then Her Husband Killed Her.

Spain has relied on an algorithm called VioGén to assess the risk of domestic violence victims being abused again and determine the level of protection they need. While the system has helped reduce repeat attacks in domestic violence cases, it has also resulted in victims being attacked again, sometimes with fatal consequences. Spain currently has 92,000 active cases of gender violence victims evaluated by VioGén, with most classified as facing little risk of being hurt again. However, a significant number of women who were assessed as low or negligible risk have reported being harmed again. At least 247 women have been killed by their current or former partners after being assessed by VioGén. The algorithm's flaws have raised concerns about the reliance on algorithms in making life or death decisions. 

‘Google says I’m a dead physicist’: is the world’s biggest search engine broken?

Google's search engine has come under scrutiny recently, with users claiming that it is not working as well as it should. The article explores the history of Google and its rise to dominance in the search market, as well as its influence over politics, social attitudes, and businesses. Critics argue that Google search has deteriorated in quality, citing issues such as spam, SEO practices, and the clutter of information boxes within search results. However, others still find Google search to be effective. The article raises questions about Google's trustworthiness and its ability to prioritize user interests.

Trump allies draft AI order to launch ‘Manhattan Projects’ for defense

Former President Donald Trump's allies are working on a comprehensive AI executive order that would establish "Manhattan Projects" to develop military technology and review regulations. The order aims to create "industry-led" agencies to evaluate AI models and protect systems from foreign adversaries. This approach differs from the Biden administration's executive order, which focuses on safety testing for AI systems. The GOP has adopted a platform that includes repealing the Biden AI executive order, claiming it hinders innovation. The framework provides insight into potential Republican policies to replace the Biden order. The greater military investment in AI could benefit tech companies like Anduril, Palantir, and Scale, which already have contracts with the Pentagon. The conservative Heritage Foundation is also drafting AI policies as part of Project 2025. Tech executives and investors, including Elon Musk and Bill Ackman, have endorsed Trump, indicating a potential second Trump administration would have a friendlier relationship with the tech industry. 

FTC is investigating how companies are using AI to base pricing on consumer behavior

The Federal Trade Commission (FTC) is investigating how companies are using AI to base pricing on consumer behavior. The agency has ordered eight companies, including Mastercard, JPMorgan Chase, and Accenture, to provide information about their AI-powered "surveillance service pricing" and its impact on privacy, competition, and consumer protection. This practice allows companies to charge different prices to different customers based on factors such as location and personal data. The FTC is concerned that this use of AI and personal data could put people's privacy at risk and wants to shed light on this "shadowy ecosystem of pricing middlemen." The investigation aims to understand the types of surveillance pricing services offered by these companies and how they are impacting consumer pricing. 

Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal

Google has become the exclusive search engine for Reddit, as other search engines like Bing and DuckDuckGo are no longer able to crawl Reddit and provide up-to-date results. This is due to Reddit's decision to lock down access to its site and prevent scraping for AI training data. Google's near monopoly on search is hindering competition and raises concerns about the quality of search results. This exclusivity is a result of a multi-million dollar deal that allows Google to scrape Reddit for data to train its AI products. 

OpenAI training and inference costs could reach $7bn for 2024, AI startup set to lose $5bn - report

OpenAI is projected to spend nearly $4 billion this year on training and inference costs, with a potential shortfall of $5 billion. The company currently uses Microsoft's servers to run inference workloads for ChatGPT, with around 290,000 servers dedicated to this task. Training ChatGPT and new models could cost up to $3 billion this year. OpenAI benefits from discounted rates from Microsoft Azure, paying about $1.30 per A100 server per hour. The company employs around 1,500 people, which could cost $1.5 billion as it continues to grow. While OpenAI generates about $2 billion annually from ChatGPT, it may need to raise additional funds within the next year to cover its losses.

Apple takes on Meta with new open-source AI model — here's why it matters

Apple has released a new open-source AI model with 7 billion parameters, signaling its commitment to the wider AI ecosystem. The model, part of Apple's DCML project, outperforms similar-sized models from Meta and Google. It is fully open source, with all weights, training data, and processes publicly available. Despite its small size and context window, the model's open-source nature makes it a significant AI release of the year. Researchers and companies can use the model to create their own small AIs without per-token costs. This aligns with the goal of creating intelligence that is affordable and accessible.

China Is Closing the A.I. Gap With the United States

At the World Artificial Intelligence Conference in Shanghai, Chinese start-up founder Qu Dongqi showcased a video created using AI technology from Chinese internet company Kuaishou. The video, which brought an old photograph to life, demonstrated the advancements China has made in the field of artificial intelligence. This technology is similar to the video generator Sora, developed by OpenAI, but the Chinese version is already available to the general public. This highlights China's progress in closing the AI gap with the United States.

Video game performers will go on strike over artificial intelligence concerns

Video game performers, including voice actors and motion capture performers, are going on strike due to concerns over AI protections. The Screen Actors Guild-American Federation of Television and Radio Artists (SAG-AFTRA) has been negotiating with major game studios for nearly two years over a new interactive media agreement. While progress has been made on wages and job safety, the two sides remain divided on the regulation of generative AI. The union is concerned that without proper safeguards, game companies could use AI to replicate actors' voices or create digital replicas of their likeness without consent or fair compensation. The strike is a last resort after exhausting other possibilities. The global video game industry generates over $100 billion in profit annually, and the performers are demanding that their work be protected. The strike will not include games covered by separate contracts that have AI protections.

AI Video Generator Runway Trained on Thousands of YouTube Videos Without Permission

A recent investigation by 404 Media has revealed that the AI video generation tool developed by Runway, a multi-billion dollar company, was trained using scraped videos from YouTube creators, brands, and pirated films. The tool, initially known as Jupiter and later released as Gen-3, received widespread praise upon its launch in June. Runway, which raised $141 million in funding last year, has not provided specific details about the training data for Gen-3. This revelation raises concerns about the ethical use of AI technology and the potential infringement of copyright laws. 

Anthropic’s crawler is ignoring websites’ anti-AI scraping policies

Anthropic's web crawler, ClaudeBot, has been scraping websites for training data for AI models without regard for the websites' anti-AI scraping policies. One of the affected websites, iFixit, noticed that ClaudeBot had accessed their content almost a million times in 24 hours, violating their Terms of Use. iFixit's CEO, Kyle Wiens, expressed concern about the unauthorized use of their content and the strain it put on their resources. Although iFixit added a crawl-delay extension to their robots.txt file to block the crawler, Anthropic claimed that their crawler can only be blocked through robots.txt. Other websites, such as Read the Docs and Freelancer.com, also reported being aggressively scraped by ClaudeBot. This is not the first time ClaudeBot's scraping activities have caused issues, as previous incidents have been reported on Reddit and the Linux Mint web forum. 

Elon Musk’s X under pressure from regulators over data harvesting for Grok AI

X (fka Twitter) is facing pressure from data regulators due to a default setting that allows users' posts to be used for training an AI chatbot called Grok. The UK and Irish data watchdogs have contacted X regarding the apparent attempt to gain user consent for data harvesting without their knowledge. Under UK GDPR, companies are not allowed to use default consent methods. The default setting on X, which comes with a pre-ticked box, allows users' posts and interactions with Grok to be used for training. Data regulators have expressed concern and emphasized the need for transparency and user notification. 

OpenAI’s Search Tool Has Already Made a Mistake

OpenAI recently announced the launch of SearchGPT, a prototype tool that uses AI to answer questions by searching the internet. However, even in the demo, SearchGPT made a mistake. When a user searched for music festivals in Boone, North Carolina in August, the top suggestion provided by SearchGPT was a fair that actually ends in July. This highlights a common issue with AI search tools, as they often exhibit errors and inaccuracies. While AI searchbots have the potential to revolutionize internet search by providing personalized answers, they still have a long way to go in terms of accuracy and reliability. 

AMD claims its top-tier Ryzen AI chip is faster than Apple’s M3 Pro

AMD recently held an event to showcase its new Strix Point Ryzen AI chips, built on the Zen 5 architecture. AMD claims that these chips can outperform Apple's M3 and M3 Pro chips, as well as beat Qualcomm's and Intel's integrated graphics. The new Ryzen AI chips offer architectural improvements, with a 16% increase in instructions per clock cycle and a 19-32% boost in graphics performance per watt. However, AMD has yet to provide concrete evidence to support these claims. The company also did not provide specific details about battery life improvements. The first laptops with the new chips will be available on July 28th. 

Papers

• Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

• AI models collapse when trained on recursively generated data

• The CLRS-Test Algorithmic Reasoning Language Benchmark

• Targeted Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs

• Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together

• Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?

• OpenDevin: An Open Platform for AI Software Developers as Generalist Agents

• AI achieves silver-medal standard in solving IMO problems

  • And some good context from Timothy Gowers

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Editor Notes

As with our last Update, there’s been AI news, but there’s also been lots of other news. I hope you’ve been staying sane and that some of you, like our wonderful editor Cole, are on a beach somewhere and not thinking about AI at all (if you’re reading this, though, that’s probably not you). 

The two podcast conversations I posted since last episode were both ones I really enjoyed: New South Wales has a very interesting and thoughtful approach to using AI in education, and David Pfau is one of the most thoughtful scientists the ML community is lucky to have. 

Also: I hate writing this probably more than you hate reading it, but it would be so, so helpful if you’d consider helping us out a bit by leaving reviews or feedback if you’re getting something from any of our content. I don’t have too many reviews on Apple Podcasts or Spotify yet, and they really help a lot :)

As always, if you want to write with us, send a pitch using this form.

News Highlight: Training AI music models is about to get very expensive

Summary 

Suno and Udio, two leading AI music startups, were sued by major record labels on June 24 — the allegation is that the companies made use of copyrighted music in their training data at massive scale. In contrast, YouTube has taken an aboveboard approach where it offers lump sums to record labels to license their catalogs for training. The outcome of this lawsuit will determine the shape of AI music generation, and, possibly, whether there is any future for it at all. 

Overview

Suno and Udio are to music what Stable Diffusion and its ilk are to images: both offer models that can, given a natural language text prompt, generate music. Unsurprisingly, the training recipe for each of the models is proprietary — and record labels allege that both startups have engaged in copyright infringement in model training and output. 

While there are similarities to the New York Times vs. OpenAI’s case, Suno and Udio appear to have even less plausible deniability than their fellow offender: while New York Times articles and their content might appear in places besides the site itself, some say it’s clear that the AI music startups must have pulled in large databases of commercial recordings. 

Furthermore, the case alleges that Suno’s and Udio’s tools are more imitative than generative: their output mimics the style of copyright-protected artists and songs. Suno and Udio both claim that they prioritize originality, and both have safeguards in place that prevent users from naming specific artists in a query, but (as you will know if you’ve spent time with any generative AI system) loopholes abound. 

According to James Grimmelmann, professor of digital and information law at Cornell Law School, there are three possible ways the case could go: the court determines, wholly in favor of the startups, that the companies did not violate fair use or imitate copyrighted works too closely; a mixed bag, where the court finds Suno and Udio did not violate fair use in training but must better control model output to avoid imitating copyrighted works; or, in favor of the record companies, the court finds issue with both the training and output of the AI models, meaning that the companies cannot train on copyrighted works without licenses or allow outputs that closely imitate copyrighted works. 

Our Take

I’m going to have an inevitably biased take on AI music in general, because I think a profound aspect of being a musician and producing music is the sensate part of it: producing orchestral music on a computer isn’t the same as sitting as a member of an actual orchestra. Yes, democratization is a thing, but the difficulty of putting something like an orchestra together and getting everything right is precisely what makes the outcome so valuable and important. All this is to say, I can personally live without AI-generated music, though I recognize the value people find in it. 

I also like the point that this Washington Post article points to: like it or not, a technology that makes it easy for you to generate and share music is going to impact the way you interact with and value music itself. Just as telegraphy and newer technologies have changed how we write and streaming services and their ilk have changed how we consume and appreciate TV shows and movies, so too would tools like Suno and Udio shift our relationship to music if they come into widespread use. I’m not entirely sure that’s a relationship to music that I want to have. 

But, I should comment on the case itself — the stakes are indeed very high for AI music companies because there aren’t so many options for music in the public domain. So it is existential for these companies. Your reaction to the verdict in this case, then, probably indicates something about your stance towards AI music and whether you think a future with it is a good future. 

On the other hand, there is also a future with licensing analogous to the deals we’re seeing between OpenAI and publishers. But, given the music industry’s power, the prices for these licensing deals is bound to be very high — only YouTube and peers with as much money would be able to cough up the cash. In this world, the future and shape of these systems once again relies on the value judgments of a handful of actors. 

—Daniel

Research Highlight: Vision language models are blind

Summary 

Researchers from Auburn University and the University of Alberta found that state-of-the-art large language models with vision capabilities (VLMs) are surprisingly poor at understanding spatial information involving basic geometric shapes, such as whether two circles overlap. They propose BlindTest, a new benchmark of 7 simple tasks that are unlikely to have prior answers in natural language on the Internet, to test VLM ability to "see" images like humans do.

Overview

Existing VLM benchmarks (such as MMMU and ChartQA) cover a wide range of subjects but the input images are not always necessary for answering the questions, i.e., answers may be inferred from the textual question and answer choices alone or memorized by the models from Internet-scale training. Motivated by this gap and inspired by visual acuity tests given to humans by optometrists, the authors design 7 low-level vision tasks that involve 2D geometric primitives. They then test four VLMs that rank the highest on existing multimodal vision benchmarks – GPT-4o, Gemini-1.5 Pro, Claude-3 Sonnet, and Claude-3.5 Sonnet. For each task, they prompt VLMs with two different questions that are semantically equivalent. The tasks and results are as follows:

1. Counting line intersections. Across 150 images of two colored lines that intersect at exactly 0, 1, or 2 points, the best accuracy is 77.33% (Sonnet-3.5) and the worst is 48.67% (GPT-4o).

2. Two circles overlapping or touching. Over 672 images of two equal-sized circles that are overlapping, tangent, or disjoint (with variations in orientation and size), the best accuracy is 92.78% (Gemini-1.5) and the worst accuracy is 72.69% (GPT-4o, again). Further, performance tends to degrade when two circles are close together.

3. Circled letter in a string. A red oval is superimposed on one of the letters in a string. The authors test three strings – Acknowledgement, Subdermatoglyphic, and a random string tHyUiKaRbNqWeOpXcZvM. Gemini-1.5 (92.81% accuracy) and Sonnet-3.5 (89.22%) outperform GPT-4o and Sonnet-3 by nearly 20 points. Except for GPT-4o, all models perform slightly better on the two English words than the random string, suggesting that knowing the word might help VLMs make better educated guesses.

4. Counting overlapping shapes. N overlapping, same-sized circles (N=5,6,7,8,9) are arranged in two rows like the Olympic logo. Performance ranges from 20.83% (Gemini-1.5) to 44.16% (Sonnet-3.5). Repeating the same arrangement of pentagons yields more disparate performances from 9.16% (Gemini-1.5) to 75.83% (Sonnet-3.5). All four models are 100% accurate in counting 5 circles while performing poorly (except Sonnet-3.5) on 5 pentagons.

5. Counting nested squares. 2 to 5 squares are nested such that each shape is entirely inside another. Sonnet-3.5 again has the best accuracy of 87.5%. GPT-4o and Sonnet-3 struggle to count even when there are only 2 or 3 squares.

6. Counting the rows and columns of a grid. VLMs struggle to count the exact number of rows and columns in an empty grid, where the best model (Sonnet-3.5) has 59.84% accuracy and the rest have 25-26% accuracy. However, adding a single word to each cell significantly improves performance for all models. For example, GPT-4o's accuracy more than doubles from 26% to 53%.   

7. Following single-colored paths. The final task asks models to count the number of unique-color paths between two given stations in a simplified subway map. "Shockingly," the authors find that no model reaches 100% accuracy even when there is only one path between two stations. Most VLMs perform worse as map complexity increases.

Overall, the BlindTest benchmark provides the first low-level, visual sanity check for VLMs. Their underwhelming performance on these simple (for humans) tasks that require zero prior knowledge are counterintuitive given their impressive performance on existing vision benchmarks, which have a data leakage problem. Addressing these limitations of VLMs is likely a non-trivial challenge and may help solve other known visual shortcomings of multimodal models such as understanding the orientation of an object.

New from the Gradient

David Pfau: Manifold Factorization and AI for Science

Listen

Dan Hart and Michelle Michael: Bringing AI to Students in New South Wales

Listen

Other Things That Caught Our Eyes

News

OpenAI illegally barred staff from airing safety risks, whistleblowers say

Whistleblowers at OpenAI have filed a complaint with the Securities and Exchange Commission (SEC), alleging that the company illegally prevented employees from raising safety concerns about its AI technology. The whistleblowers claim that OpenAI issued overly restrictive employment, severance, and nondisclosure agreements that could have penalized workers who reported concerns to federal regulators. These agreements violated federal laws protecting whistleblowers and their right to anonymously disclose damning information about their company. The complaint comes amid concerns that OpenAI prioritizes profit over safety in developing its technology. The SEC has not yet confirmed whether an investigation has been launched.

Data workers detail exploitation by tech industry in DAIR report

A new report by the Data Workers' Inquiry, a collaboration between AI ethics research group DAIR and TU Berlin, sheds light on the exploitation of data workers in the tech industry. The report highlights the hidden labor of data work, such as moderation and annotation, which is often outsourced to poorer countries where workers are paid significantly less than their American or European counterparts. The conditions of this work, although not physically dangerous, can be psychologically damaging. The reports, which are largely anecdotal, provide firsthand accounts of the challenges faced by data workers, including mental health issues and a lack of support from employers. The report emphasizes the need for companies to address the exploitation of data workers and calls for further research on the topic.

New York Times Experiments With a New Headline Writer: OpenAI

The New York Times has been experimenting with OpenAI's generative AI technology to develop a tool that can generate headlines for articles and apply the newspaper's style guide. The leaked code revealed that the Times used OpenAI to create a headline writer that could potentially replace editors. While the project was only an early experiment and not used by the newsroom, it highlights the increasing use of AI in newsrooms. However, there are concerns that AI could lead to further job losses for journalists. The Times is currently involved in a lawsuit against OpenAI and Microsoft for alleged copyright infringement.

Deepfake Creators Are Revictimizing GirlsDoPorn Sex Trafficking Survivors

Deepfake creators have reached a new low by using videos of sex trafficking victims as the basis for nonconsensual deepfake pornography. An account on a deepfake sexual abuse website posted 12 celebrity videos that were created using footage from GirlsDoPorn, a sex trafficking operation. The videos, which were up to 21 minutes long, had celebrity faces added using AI. Deepfake technology has become increasingly realistic and accessible, leading to the proliferation of websites and apps designed for deepfake sexual abuse, while laws to protect victims and limit the use of these tools are lagging. Legal proceedings against the creators of GirlsDoPorn and affiliated individuals are ongoing, with survivors being awarded damages and copyright ownership of the videos.

The A.I. Boom Has an Unlikely Early Winner: Wonky Consultants

The rise of AI has created a demand for consultants who can help businesses understand and implement this technology. Companies like Boston Consulting Group and McKinsey & Company are experiencing a surge in revenue and hiring as they assist businesses in navigating the implications of AI and how it can benefit their operations. For example, Reckitt Benckiser, the maker of Lysol and Mucinex, sought the expertise of Boston Consulting Group to explore how AI could be applied to their business. 

Humane execs leave company to found AI fact-checking startup

Former employees of Humane, a struggling AI hardware company, have left to start their own startup called Infactory. Infactory is a fact-checking search engine that aims to provide accurate information by pulling data directly from trusted sources, rather than relying on generative AI services. The founders, Brooke Hartley Moy and Ken Kocienda, emphasize the importance of using AI selectively and focusing on computational and factual data. Infactory plans to target enterprise customers, such as newsrooms and research facilities, and will initially focus on data-related subjects. The startup has raised pre-seed funding and will be seeking seed funding in the next six to 18 months. Despite their departure from Humane, the founders deny that their decision was a direct result of the company's struggles. Infactory is set to launch in a few months.

The AI-focused COPIED Act would make removing digital watermarks illegal

A new bill called the Content Origin Protection and Integrity from Edited and Deepfaked Media Act (COPIED Act) has been introduced by a bipartisan group of senators — it aims to authenticate and detect artificial intelligence-generated content, protecting journalists and artists from having their work used without permission. The Act directs the National Institute of Standards and Technology (NIST) to create standards and guidelines for proving the origin of content and detecting synthetic content through watermarking. It also requires AI tools to allow users to attach information about the origin of creative or journalistic content and prohibits the removal of this information. Content owners can sue companies that use their materials without permission or tamper with authentication markers. The bill has received support from publishing and artists' groups. 

Why The Atlantic signed a deal with OpenAI

The Atlantic has signed a deal with OpenAI, allowing the AI company to use The Atlantic's archives as training data. The CEO of The Atlantic, Nicholas Thompson, explains that the deal provides revenue and a potential traffic source for the magazine. The deal includes three main components: allowing OpenAI to train on The Atlantic's data for two years, a product partnership where OpenAI provides credits and potential engineering support, and the inclusion of The Atlantic in OpenAI's search product. The goal of the deal is to shape the future of AI and ensure that journalists and media companies are paid for their work.

AI Has Become a Technology of Faith

The article discusses the launch of a new company called Thrive AI Health, which aims to bring OpenAI's technology into the healthcare industry. The company plans to develop a hyper-personalized AI health coach that will generate personalized insights based on a user's biometric and health data. The article raises concerns about privacy and the potential misuse of personal health information. The founders of Thrive AI Health argue that people are willing to share personal details with AI language models and that the technology can offer behavioral solutions to improve health outcomes. However, the article questions the feasibility and potential risks of such a product. 

China's homegrown OS fires back at AI PCs — openKylin gets AI assistant, text-to-image generation, and local LLM support

China's openKylin operating system, an open-source OS based on Linux, has released a new version that is deeply integrated with AI. The OS features support for on-device LLMs, an AI assistant, and text-to-image generation. The aim is to boost productivity and user experience for those using domestic operating systems. AI PCs, equipped with advanced processors capable of running generative AI tasks locally, have gained popularity in China. Lenovo sees China as a unique market for AI PCs due to data-localization requirements. OpenKylin is part of China's effort to decrease dependence on foreign operating systems. However, Windows remains the dominant OS in China with nearly 80% of the market.

Microsoft and Apple ditch OpenAI board seats amid regulatory scrutiny

Microsoft and Apple have both stepped down from their board seats at OpenAI, a nonprofit organization focused on artificial intelligence (AI) research. Microsoft had secured a non-voting seat less than eight months ago, but concerns over antitrust issues have led to the decision. OpenAI will now adopt a new approach to engage with strategic partners and investors, including Microsoft and Apple, through regular stakeholder meetings. The move comes as UK and EU regulators are investigating Microsoft's partnership with OpenAI, along with other AI deals involving major tech companies. Microsoft has invested over $10 billion in OpenAI, making it the exclusive cloud partner and benefiting from OpenAI's AI models to enhance its own products and services. 

AI Needs Copper. It Just Helped to Find Millions of Tons of It.

KoBold Metals, a company in California, has used AI-driven technology to discover a large copper deposit in Zambia. The company estimates that the mine could produce at least 300,000 tons of copper per year, worth billions of dollars annually. An independent assessment largely corroborated the size of the deposit. KoBold expects the value of the mine to increase as they continue to map the full extent of the highest-grade ore. 

Japan’s Defense Ministry unveils first basic policy on use of AI

The Defense Ministry of Japan has released its first basic policy on the use of AI in order to address concerns about recruitment and the need to utilize personnel more efficiently. With a declining and aging population, Japan aims to leverage AI to overcome these challenges and keep up with China and the United States in terms of military applications of the technology. The policy highlights the potential of AI to alleviate manpower shortages and enhance the capabilities of the Self-Defense Forces. Japan’s move reflects its recognition of the importance of AI in maintaining its technological edge and addressing demographic issues.

China’s AI competition deepens as SenseTime, Alibaba claim progress at AI show

Chinese AI companies SenseTime and Alibaba showcased their progress in developing LLMs at the World Artificial Intelligence Conference (WAIC) in Shanghai. SenseTime released its latest foundational model, SenseNova 5.5, which boasts a 30% improvement in performance compared to the previous version. SenseTime claimed that SenseNova 5.5 outperforms GPT-4o in five out of eight key metrics. Alibaba's cloud computing unit highlighted the growth of its Tongyi Qianwen LLMs, with downloads doubling to over 20 million in the past two months. The number of customers served by Alibaba Cloud Model Studio also increased by over 150% to 230,000. The competition in the Chinese LLM market is intensifying, with only a few companies predicted to dominate in the future. 

Papers

Our recs:

• Universal Length Generalization with Turing Programs

• MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

• Learning to (Learn at Test Time): RNNs with Expressive Hidden States

• Me, Myself, and AI: The Situational Awareness Dataset for LLMs

• Distilling System 2 into System 1

• Not a paper, but a report: Othello-GPT learned a bag of heuristics

• Selective Perspectives: A Content Analysis of The New York Times’ Reporting on AI

• Faux Polyglot: A Study on Information Disparity in Multilingual LLMs

• FlashAttention-3

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

News Highl…

Read more

Editor Notes

Good morning. Not a terrible amount to call out this week. If you’re in the states, I think other events have seized everyone’s attention recently.

As always, if you want to write with us, send a pitch using this form.

News Highlight: How AI is accelerating Candy Crush development

Summary 

In a recent interview with gamesindustry.biz, the leads of Activision Blizzard King’s (AI) AI Labs discussed the role that AI has played throughout the development of “Candy Crush”, one of their largest and most successful games. The scientists discuss their goals in building a playtesting bot as well as its usage by developers to increase both the quality and speed of level development. Additionally, the researchers at King emphasize the importance of humans working in synch with these AI tools rather than being replaced by them. This is a welcome contrast to the dominant narrative of AI in gaming which typically consists of either the theft of artists’ intellectual property by generative models or further weakening the job market for games and software developers.

Overview

Since April 2012, developers at ABK have released an average of over 4 new Candy Crush levels every day for the past 12 years for a total of almost 17 thousand unique levels. This volume of content is practically unprecedented in the industry and much of its recent throughput could be attributed to ABK’s acquisition of the AI firm Peltarion in 2022. After the acquisition, developers from Peltarion began to flesh out leadership positions in King’s AI labs and develop plans for playtesting bots. By training neural networks using nothing but their players' game data, the scientists at King’s AI labs were able to build playtesting bots at a variety of human skill levels. These bots are used by the developers at King to automatically detect bugs before release and helped them reduce the number of manual tweaks by over 95%. Additionally, these bots can be used by game designers to understand the appropriateness of the intended difficulty of the levels as well as the impact of shuffle (randomization), a core gameplay mechanic of candy crush. By investing in these AI development tools, game designers and quality assurance (QA) workers at ABK have been able to supercharge their outputs, developing more levels faster, as well as better understanding how their players will interact with a level (and its difficulty beforehand) .

While some may see these AI tools as a path towards generating more things faster and reducing pesky labor costs, the developers at ABK consistently emphasized the impact and importance of the human designers working in tandem with AI tools. While these AI solutions may automate away some of tasks that are typical of designers and QA workers, it can free their responsibilities from some of the more menial, repetitive, and tedious tasks so that they can focus on more meaningful and high impact tasks. Ultimately, the designers (and other humans working in game development) are the ones who know what “fun” really is and there is no expectation that generative AI will ever be able to replicate or develop that kind of intuitive sense.

Our Take

When I first started working gaming, I got an inside look at how playtesting bots can supercharge development and improve the quality of work for all kinds of game developers (designers, QA, data scientists, product leads etc). Seeing that body of work being pursued by other gaming companies has me super enthused. However, as excited as I am by the end results, I wish the scientists ABK shared more details on the inner workings of their models. While I am sure there is a businessperson making a case to not reveal any trade secrets, it would have been nice to get really any details on how they trained and tested their models. Are they using reinforcement learning? What about transformers? Hopefully there will be a GDC talk or paper coming soon where we could learn more! - Justin 

Research Highlight: Just How Flexible are Neural Networks in Practice?

Summary 

Neural networks are commonly believed to fit training sets with as many samples as they have parameters. However, the effectiveness of this fitting depends significantly on the training procedures such as the optimizer or regularizer and the specific architecture used. In this work, the authors study how flexible neural networks truly are when it comes to fitting data. The key findings are as follows:

1. Conventional optimizers typically locate minima where models fit far fewer samples than the models parameter count;

2. Convolutional neural networks (CNNs) surpass Multi-Layer Perceptrons (MLPs) and Vision Transformers (ViTs) in parameter efficiency with randomly labeled data;

3. Stochastic gradient descent finds solutions that fit more training data than full-batch gradient descent;

4. ReLU activation functions enhance data fitting capabilities by mitigating vanishing and exploding gradients.

Overview 

It is widely believed that a neural network can fit a training set with at least as many samples as it has parameters, which forms the basis for understanding overparameterized and underparameterized models. Despite the theoretical guarantees on neural networks being universal function approximators, the models we train in practice have limited capacities and often lead to suboptimal local minima during training, constrained by the methods we use. Their flexibility to fit data varies based on a number of factors such as the characteristics of the data, the architecture and size of the model or the optimizer used. In the recent paper ‘Just How Flexible are Neural Networks in Practice?’, the authors empirically study this relationship. 

To quantify this flexibility of a model, the authors use Effective Model Complexity (EMC) metric, which estimates the maximum number of samples a model can perfectly fit. The EMC is determined through an iterative training process, where the model is trained on progressively larger datasets until it fails to achieve 100% accuracy. The study uses different initializations and sample subsets in each iteration to ensure fairness in evaluation. An important factor when studying fitting of a model to data is also to ensure the model is not under trained on a given data. The authors verify this by checking for the absence of negative singular values in the Hessian loss. 

To comprehensively dissect the factors influencing neural network flexibility, a variety of models with varying depths and widths such as  Multi-Layer Perceptrons (MLPs), Convolutional neural networks (CNNs) , EfficientNet, and Vision Transformers (ViTs) on a range of datasets including ImageNet. Multiple optimizers are also used to account for the effect of stochasticity and preconditioning on the minima.  Below we analyze some key empirically established relationships and trends across different training or dataset choices and model fitting.

Effect of Dataset

As shown in the figure above, the findings indicate considerable differences in the Effective Model Complexity (EMC) of networks depending on the type of data used for training. For example, networks trained with tabular data (Income, Forest, Covertype) demonstrate greater capacity. Further for image classification datasets, there is a notable correlation between test accuracy and EMC. 

Optimizers and Model architectures

As seen from the figure above (left), CNNs demonstrate greater parameter efficiency compared to MLPs and ViTs, even when trained on images with random labels. This efficiency cannot be explained by their inductive bias, which suggests that CNNs generalize effectively due to their preference for labeling functions that exhibit specific symmetries. It is commonly believed that stochastic training acts as a regularizer. However, as shown in the Figure above (right), SGD tends to locate minima that fit more training data compared to full-batch gradient descent. 

Generalization

One can observe from the figure above that the disparity in a model's ability to fit correctly versus incorrectly labeled samples serves as a predictor of its generalization capabilities. Specifically, models that exhibit strong generalization can accommodate significantly more correctly labeled samples than randomly labeled ones, indicating their ability to generalize effectively within their training set.

Our Take

We often assess whether a neural network is overparameterized or underparameterized by counting its parameters. In practice, the amount of data a network can fit provides a more accurate measure of its capacity. I find this paper interesting since it provides empirical evidence on the interplay between model architecture, training procedures, and data characteristics which is complementary to the theoretical guarantees in this space. Further, they indicate that neural networks exhibit parameter inefficiency, and might help to develop new parameterizations that could enhance their overall efficiency. However, I would like to see more apples to apples comparison in certain ablations such as comparing performance across datasets like CIFAR10, ImageNet despite them having different classes and characteristics. 

– Sharut

New from the Gradient

Kristin Lauter: Private AI, Homomorphic Encryption, and AI for Cryptography

Listen

Sergiy Nesterenko: Automating Circuit Board Design

Listen

Other Things That Caught Our Eyes

News

ChatGPT is hallucinating fake links to its news partners’ biggest investigations

ChatGPT is generating fake URLs and directing users to 404 errors instead of real article pages for several major news media companies that have partnered with OpenAI. These publications include The Associated Press, The Wall Street Journal, The Financial Times, The Times, Le Monde, El País, The Atlantic, The Verge, Vox, and Politico. Despite the licensing deals stating that ChatGPT will produce attributed summaries and link to the publications' websites, the chatbot is currently unable to reliably link to these partner publications' most noteworthy stories. OpenAI has acknowledged that the citation features promised in the licensing contracts are still in development and not yet available in ChatGPT. The article highlights the concern among journalists about ChatGPT's potential as a search tool and the need for transparency and protections for the integrity of journalism. 

McDonald's is removing its AI drive-thru voice-ordering system from over 100 restaurants after its mishaps went viral

McDonald's is removing its AI drive-thru voice-ordering system, called the Automated Order Taker, from over 100 restaurants after videos showcasing flaws with the technology went viral. The fast-food chain collaborated with IBM in 2021 to develop and deploy the AI software. However, the technology did not meet expectations, leading to its removal. This decision marks the end of a test period conducted with IBM. The mishaps with the AI system highlight that generative AI is not yet advanced enough to fully replace human jobs in industries like restaurants. 

How This Real Image Won an AI Photo Competition

Photographer and writer Miles Astray submitted a photograph titled "F L A M I N G O N E" to the 1839 Awards, a photography competition with a category for images created by artificial intelligence (AI). The image appeared to show a flamingo without its head and neck, but it was actually a real photograph taken by Astray. The photograph won the People's Vote Award before being disqualified. Astray wanted to show that nature can still outdo machines in terms of creativity and beauty. He also highlighted the dangers of AI-generated content being indistinguishable from real content. Astray believes that tagging AI-generated images and videos and educating people to be critical thinkers can help address the potential for AI-generated misinformation. 

Copilot dreams and nightmares: Microsoft insiders share what they really think about the company's AI future

Microsoft insiders have shared their thoughts on the company's AI future and its partnership with OpenAI. While there is optimism about Microsoft's ability to improve its AI offerings and leverage its existing trust with customers, there are concerns about the value of new AI services for corporate customers and the resentment caused by the OpenAI partnership. Microsoft's Copilot offerings, based on OpenAI's GPT models, aim to provide automated support for various tasks. However, there are mixed early results and a race to add value before customers question the return on investment. Microsoft is also facing pressure from OpenAI's move into the business market. 

Waymo One is now open to everyone in San Francisco

Waymo, the autonomous vehicle company, has announced that its Waymo One ride-hailing service is now available to everyone in San Francisco. Waymo has been operating in the city for several years, gradually scaling its service. The Waymo One service offers safe, sustainable, and reliable transportation, with about 30% of rides being to local businesses. Waymo's fleet is all-electric and sources 100% renewable energy. The service has helped reduce carbon emissions and improve personal safety for riders. Waymo also offers a unique way for tourists to experience the city. With a focus on safety, Waymo has a track record of over 20 million rider-only miles and is committed to working with city and state officials to ensure responsible growth. 

Training AI music models is about to get very expensive

The article discusses the potential legal implications and financial consequences for AI music companies that use copyrighted works to train their models. Record labels have filed lawsuits against these companies, alleging that their models imitate copyrighted works too closely. The outcomes of the lawsuits could range from the court finding in favor of the AI startups, determining that they did not violate fair use, to the court finding fault on both the training and output sides of the AI models, resulting in potential damages for infringement. The article also mentions the possibility of a future licensing market for AI music models, similar to what has already happened with text generators. However, licensing deals could be costly, potentially limiting access to powerful music models to those with significant financial resources. The article concludes by noting that training AI models exclusively on public domain music would be challenging and result in models that are far less impressive than what exists today.

MIT robotics pioneer Rodney Brooks thinks people are vastly overestimating generative AI

Rodney Brooks, a robotics pioneer and co-founder of companies like Rethink Robotics and iRobot, believes that people are overestimating the capabilities of generative AI. While he acknowledges that generative AI is impressive technology, he argues that it cannot perform all tasks that humans can and that humans tend to overestimate its abilities. Brooks emphasizes that generative AI is not human-like and should not be assigned human capabilities. He provides an example of a warehouse robotics system, where using generative AI to tell the robots where to go would actually slow things down compared to connecting them to a stream of data from the warehouse management software. Brooks also highlights the importance of solving solvable problems and designing robots for practical purposes rather than building human-like robots. He believes in making technology accessible and purpose-built, and he cautions against the mistaken belief that technology always grows exponentially. While he sees potential for LLMs in domestic robots for specific tasks, he emphasizes that the problem lies in control theory and math optimization, not in language capabilities. 

Here comes a Meta Ray-Bans challenger with ChatGPT-4o and a camera

Solos, a competitor to Ray-Ban Meta Smart Glasses, plans to release a camera-equipped version called Solos AirGo Vision later this year. These smart glasses will feature OpenAI's new GPT-4o AI model, allowing the camera to recognize objects and answer questions about what the wearer sees. The glasses will have a swappable frame system, allowing users to switch out the camera for different looks or sun shades. The Vision will also include notification LEDs for incoming calls or emails and can be integrated with Google Gemini and Anthropic's Claude AI models. While the price and release date for the AirGo Vision have not been announced, they are expected to cost more than $249.99, the price of the camera-less version. The Ray-Ban Meta Smart Glasses currently start at $299.

Y Combinator rallies startups against California’s AI safety bill

Y Combinator and several AI startups have expressed opposition to California's Senate Bill 1047, which aims to regulate the development of AI systems. The bill would require developers of large AI models to take precautions such as safety testing, implementing safeguards, and post-deployment monitoring. Y Combinator argues that the responsibility for misuse should lie with those who abuse the tools, not the developers. They believe that holding developers liable for unintended misuse could stifle innovation and discourage investment in AI research. The letter also criticizes other aspects of the bill, including the proposed requirement for a 'kill switch' to deactivate AI models. Y Combinator claims this would impact the development of open-source AI. 

Google is bringing Gemini access to teens using their school accounts

Google is expanding access to Gemini to teen students using their school accounts. This move aims to prepare students for a future where generative AI is prevalent. Gemini provides real-time feedback to help students learn more confidently. Google assures that it will not use data from student chats to train its AI models and has implemented guardrails to prevent inappropriate responses. The company recommends that teens use the double-check feature to develop critical thinking skills. Gemini will be available in English in over 100 countries and will be off by default for teens until administrators enable it. Additionally, Google is launching the Read Along in Classroom feature globally, which helps students improve reading skills with real-time support. Educators will have access to tools for creating, managing, and sharing interactive lessons, as well as the ability to mark assignments and perform bulk scoring actions. 

Stability AI lands a lifeline from Sean Parker, Greycroft

Stability AI has secured new funding from investors including Sean Parker, Greycroft, and Coatue Management. The company has not disclosed the amount raised. Stability AI has faced financial difficulties, with unpaid cloud bills and a lower valuation for the startup. However, the new investors have committed $80 million to take over Stability and have negotiated debt forgiveness and the release from future obligations. Stability AI plans to focus on growing its managed image, video, and audio pipelines, building custom enterprise models and content production tools, and delivering APIs for consumer apps. The company aims to remain committed to open-source principles and continue developing powerful generative AI models.

OpenAI buys a remote collaboration platform

OpenAI has acquired Multi, a startup that developed a video-first collaboration platform for remote teams. The deal is primarily an acqui-hire, with most of Multi's team joining OpenAI. Multi offered features such as collaborative screen sharing, customizable shortcuts, and automatic deep links for code, designs, and documents. This acquisition aligns with OpenAI's strategy of investing in enterprise solutions, as they have been focusing on developing AI-powered tools for businesses. OpenAI's revenue is projected to exceed $3.4 billion this year. The acquisition of Multi suggests that OpenAI may expand its offerings to include videoconferencing and remote collaboration capabilities in the future.

YouTube is trying to make AI music deals with major record labels

YouTube is seeking to make deals with major record labels, including Universal Music Group, Sony Music Entertainment, and Warner Records, to license their songs and train its AI music tools. The platform aims to use the licensed music to develop new AI tools that will be launched later this year. YouTube has not disclosed the exact fee it is willing to pay for these licenses, but it is expected to be a one-time payment rather than a royalty-based arrangement. However, convincing both artists and labels may be a challenge, as Sony Music has warned against unauthorized use of its content, and UMG has previously pulled its music catalog from TikTok due to concerns about AI-generated music. In January, over 200 artists called for tech companies to stop using AI to infringe upon the rights of human artists. 

Opera’s browser adds AI-powered image generation and better multimedia controls

Opera is releasing the second version of Opera One in developer beta, featuring new multimedia controls, split tabs, and AI capabilities. The new version allows users to have floating multimedia controls that can be resized and match the browser's theme. It also introduces split tab windows, enabling users to work on two web pages simultaneously. Additionally, Opera One R2 includes AI features such as a page context mode that allows users to ask questions about a web page and AI-powered summarization. Opera plans to release Opera One R2 to a wider user base later this year.

Papers

Once again, a list:

• How Do LLMs Acquire Factual Knowledge During Pretraining?

• Refusal in LLMs is Mediated by a Single Direction

• Transcoders Find Interpretable LLM Feature Circuits

• The Surveillance AI Pipeline

• Just How Flexible are Neural Networks in Practice?

• Evaluating the World Model Implicit in a Generative Model

• Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces

• Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data

• Detecting Hallucinations in LLMs Using Semantic Entropy

• Linguistic inputs must be syntactically parsable to fully engage the language network

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Read more

Editor Notes

Good morning. 

Leopold Aschenbrenner is ubiquitous and OpenAI has a deal with Apple. 

Our highlights today have absolutely nothing to do with either of these items. 

A few great pieces from around town:

• The Work of Art ()

• Girlfriends, Inc. ()

• It’s Time to Listen to Teachers About AI! ()

As always, if you want to write with us, send a pitch using this form.

News Highlight: AI-Paraphrased News Cannibalizes Original Journalism

Summary 

BNN Breaking, a site that churned out AI-paraphrased versions of news articles, shut down after two years as numerous complaints about its content stealing and fabrication came to light. The dubious news outlet had a licensing agreement with Microsoft, which owns MSN. Its articles were also picked up by prominent Western outlets such as The Washington Post and The Guardian and were frequently surfaced on Google News. Individuals suffered reputation damage due to the falsehoods BNN published while local journalism organizations lost already scarce revenue to BNN's plagiarism.

Overview

As detailed in a recent New York Times article, BNN Breaking, which looked like a reliable news site, was in reality an AI chop shop. It was founded in Hong Kong by Gurbaksh Chahal, a serial entrepreneur who had a criminal history of domestic violence, and employed dozens of freelance journalists based in countries like Pakistan, Egypt, and Nigeria. Employees uploaded articles from other news sites to a generative AI tool and were asked to manually "validate" the paraphrased results. Eventually, the tool churned out hundreds or even thousands of stories a day and randomly assigned journalists' bylines.

One of the BNN articles promoted on MSN.com falsely associated Irish DJ and talk-show host Dave Fanning, whose photo was featured, with a different Irish broadcaster accused of sexual misconduct. The name of the broadcaster facing trial was not released in the original article and the AI "presumably paired the text with a generic photo of a 'prominent Irish broadcaster'", according to the India-based journalist whose byline was on the story. Fanning filed a defamation lawsuit against Microsoft and BNN Breaking.

BNN's operations also harm smaller, local news outlets who already struggle to get a share of advertising dollars to support their work. For example, its AI-rewritten version of an original article published by South African outlet Limpopo Mirror ranked higher in Google search than the genuine version. Only after Google rolled out an update targeting "spammy" sites in March did BNN's stories stop showing up in search results.

Our Take

I was quite shocked that a source that seems so obviously suspicious (perhaps only in hindsight) gained as much traction as it did and lasted as long as two years. The NYT article mentions that content curation on MSN.com was increasingly carried out by AI instead of human editors, and I can't help thinking about the layers of algorithmic decision making that shapes what we see on a daily basis, from what is surfaced in search to the actual content. Does AI have a bias for AI-generated content?

What makes BNN intriguing, and what I think partially contributed to its success before its downfall, was that real human journalists were involved, and at least some of them bought into the promise of a "revolution in the journalism industry" before they realized their own reputation and integrity were being jeopardized. At the same time, BNN's workforce — a scattered network of relatively inexperienced journalists based in developing countries — is reminiscent of the kind of labor exploitation familiar to tech. How big is the impact of AI pink slime and AI-generated news that appear to be of high quality, in a world where human writers are wrongly accused of relying on AI and are being replaced by AI anyway? I'd like to think we as consumers of information all have something within our control that we can do in the service of truth.

– Jaymee

I want to echo and spend a minute on Jaymee’s last point: that we have something within our control that we can do in the service of truth, even as consumers. I think often of L.M. Sacasas’s point—made by others as well, but especially well by him—that we can conflate what feels technologically “advanced” for what is best for human beings as we are. This manifests in different ways, and BNN provides perhaps one example of when we believe too deeply in the logic of inevitability. News organizations have struggled with these questions and laid out guidelines and standards for the use of AI in the newsroom, but there clearly isn’t something like “societal agreement” on what’s acceptable and good for us. Will we ever make different decisions? 

—Daniel

Research Highlight: Open-Endedness is Essential for Artificial SuperIntelligence

Summary 

This position paper out of Google DeepMind formalizes the concept of an “open-ended” AI system and argues that achieving Artificial Superhuman Intelligence (ASI) will require the development of such systems. The authors propose an observer-dependent and temporal definition that a system is “open-ended” so long as the sequence of artifacts it produces is both learnable and novel to the observer. Learnable artifacts are those that make future artifacts more predictable and novel artifacts are those that are unpredictable given previously generated artifacts. Novelty guarantees information gain while learnability ensures that the information gain is in some sense “meaningful” and that the artifact is interesting to the observer. They argue that we have yet to develop any general “open-ended” systems, describe how their framework maps to existing SoTA AI systems, and offer four overlapping paths towards open-ended foundation models.

Overview 

The authors begin by claiming that because:

1. “open-ended invention is the mechanism by which human individuals and society at large accumulates new knowledge and technology” and,

2.  an ASI could, by definition, perform a wide range of tasks at a level beyond any human’s capability,

It must follow that open-endedness is an essential property for ASI. Here “essential” means something along the lines of necessary but not sufficient. They describe an open-ended system as “an autonomous system [that] self-improves towards increasingly creative and diverse discoveries without end.”

In their view, scaling foundation models alone won’t get us open-endedness. They note the limited supply of high-quality textual and visual data, and conjecture that “the trend of improving foundation models trained on passive data by scaling alone will soon plateau”. Something more targeted towards open-endedness is needed to discover new knowledge. “Open-ended” is pretty hand-wavy, so in order to catalyze research into open-endedness by providing clarity, they define an open-ended system as one that produces novel and learnable artifacts, then formalize these concepts in the following way:

With this setup, they define the system as novel if the artifacts being generated become increasingly unpredictable with respect to the observer’s model at some fixed time t:

And the system is learnable if conditioning on a longer history makes future artifacts more predictable in the sense of a lower loss value:

Novelty and learnability are necessary in tandem: without novelty, the observer is able to predict all the artifacts and there is no new information being gained; without learnability, the observer can’t make tractable process (the authors give the example of a TV-set that generates white noise—endlessly novel in the sense of the unpredictability of its aleatoric uncertainty but not interesting or learnable in any sense. 

This definition is very general by design. Leaving the loss metric $\ell(t,T)$ undefined allows leeway in measuring the “interestingness” of artifacts. And by incorporating time and making novelty and learnability observer-dependent, this definition allows for the possibility of systems that are only temporarily open-ended or only open-ended to certain observers.

The authors relate how their definition of open-endedness maps to existing AI systems. They highlight three archetypal AI systems that do exhibit open-endedness but only in a narrow domain:

1. Systems that augment RL with self-play (e.g. AlphaGo, AlphaGo and similar systems for StarCraft, Stratego, DotA, and Diplomacy)

2. Systems that use unsupervised environment design (UED) to establish an automatic curriculum (e.g. AdA, a large-scale agent capable of solving tasks in a 3d environment)

3. Systems that employ evolutionary algorithms to train agents (e.g. Paired Open-Ended Trailblazing (POET), which trains a population of agents across an evolving range of environments)

And what about current SoTA foundation models? The authors classify these as general but not open-ended. They argue that since these models are trained on fixed datasets, it follows that if the distribution of this data is learnable then these models cannot be endlessly novel. Further the distribution of this data must be learnable because the foundation model learned it in the first place!

But all hope is not lost for foundation models—the authors contend that augmenting foundation models with open-endedness is a viable route toward ASI. They offer four overlapping paths to such open-ended foundation models:

1. RL: RL is fundamental to the existing narrow open-ended AI systems and there is already promising research into using RL on top of foundation models. In particular foundation models could overcome the difficulty of guiding RL algos in high-dimensional domains by serving as a sort of proxy observer (e.g. Voyager, Motif, OMNI)

2. Self-Improvement: self-improvement loops wherein the model critiques itself that hopefully allow the system to “generate new knowledge…beyond the human curated training data” (e.g.  Anthropic’s Constitutional AI, OpenAI’s work on self-critiquing models, SELF-INSTRUCT)

3. Task generation: “adapting the difficulty of tasks to an agent’s capability so that they remain forever challenging yet learnable” (e.g. WebArena, Interactive Real World Simulators)

4. Evolutionary Algorithms: foundation models can be used to generate selection and mutation operators (e.g. ELM, 2024 survey paper of evolutionary algos for LLMs)

Our Take

Generative AI—in particular open-ended GenAI or GenAI for discovery—surfaces a fascinating tension between novel outputs and coherent outputs. I think this tension tracks well with the dual requirements of novelty and learnability introduced in this paper. In some sense it feels misguided to expect novelty from a system that’s trained with the sole objective of minimizing surprise. A good illustration of this tension is decoding open-ended text from an LLM. Simply generating the most-likely next token or using some sort of greedy beam search is a bad strategy! It tends to yield uninteresting and repetitive outputs. I love this plot from Holtzman et al. 2020 comparing beam search next-token probabilities to the probabilities the same LLM assigns to a human-generated text (one that it hasn’t seen during training):

The beam search probabilities are high by design—what’s…surprising…is how much the probabilities of the human-generated text jump around. One solution is to introduce some randomness into the text generation: rather than generating the most-likely next token, sample from the next-token distribution. But pure sampling has a tendency to go off the rails resulting in incoherent (if not also beautiful and poetic) gibberish:

Sampling promotes novelty but too much of it harms learnability. Common alternatives to pure sampling like top-k sampling or nucleus sampling aim for a middle ground by lopping off the tail end of the probability distribution before sampling. This avoids generating very unlikely tokens and helps keep the LLM on track while also allowing for diverse outputs. But it feels sort of hacky. More recently, the cutting edge in generation involves incorporating some sort of Monte Carlo tree search over outputs—often optimizing over some metric other than probability.

Open-ended natural language generation is particularly tricky because the success criteria are not very clearly defined. Domains like games, protein folding, or geometry are more tractable because success criteria are clearer and valid outputs can be verified automatically. Successful systems like AlphaFold and AlphaGeometry combine an LLM architecture for generating ideas with some sort of “external verifier” (e.g. AlphaFold’s structure module that enforces stereochemical constraints or AlphaGeometry’s rule bound deduction engine).

Interestingly, the authors of this open-endedness paper more or less elide the foundation model + external verifier paradigm. Neither AlphaFold nor AlphaGeometry is mentioned or cited in the paper (although one of the authors references AlphaFold in a tweet about this paper as revolutionary but not fully general). Instead, they emphasize a foundation-models-as- “proxy observer”-for-a-RL-or-evolutionary-search-algorithm paradigm. There are though good reasons to be skeptical about the use of foundation models as verifiers.

– Cole

I want to at least mention this figure from David Pfau (source):

The point being to note that open-endedness—as the authors see it—doesn’t seem to fit neatly into the right side of Pfau’s diagram. We can confirm this by way of the examples the authors provide for open-endedness: AlphaGo (deep RL) is a positive example in that it produces policies that are novel to human expert players; contemporary foundation models are a negative example (the distribution of their data is learnable, and cannot be endlessly novel). 

Anyway, I’m still coming to conclusions on all this and would love to hear thoughts. 

—Daniel

New from the Gradient

C. Thi Nguyen: Values, Legibility, and Gamification

Listen

Vivek Natarajan: Towards Biomedical AI

Listen

Other Things That Caught Our Eyes

News

States Take Up A.I. Regulation Amid Federal Standstill

Lawmakers in California are pushing for new regulations on AI to protect consumers and jobs. The proposed bills aim to impose the strictest restrictions on AI in the nation, addressing concerns such as job displacement, disinformation in elections, and national security risks. The measures include rules to prevent AI tools from discriminating in housing and healthcare services, as well as protecting intellectual property and jobs. With the federal government slow to act on AI regulation, state legislators are taking the lead, with California proposing the most bills at 50. These state regulations could set a precedent for the entire country. 

First Came ‘Spam.’ Now, With A.I., We’ve Got ‘Slop’

"Slop" is a term used to describe shoddy or unwanted AI content in various online platforms, including social media, art, books, and search results—it refers to inaccurate or irrelevant suggestions or information generated by AI systems. For example, when Google suggests adding nontoxic glue to make cheese stick to a pizza or when a digital book seems similar to what you were looking for but falls short. The term gained more attention when Google incorporated Gemini into its US-based search results, where it attempted to provide an "AI Overview" at the top of the results page. 

U.S. Clears Way for Antitrust Inquiries of Nvidia, Microsoft and OpenAI

The US Justice Department and Federal Trade Commission have reached a deal to proceed with antitrust investigations into the dominant roles of Microsoft, OpenAI, and Nvidia in the AI industry. The Justice Department will investigate Nvidia, the largest maker of AI chips, for potential antitrust violations, while the FTC will examine the conduct of OpenAI, the creator of the ChatGPT chatbot, and Microsoft, which has invested heavily in OpenAI. This agreement reflects the increasing regulatory scrutiny of AI technology and its potential impact on jobs and society. The Justice Department and FTC have been actively working to curb the power of major tech companies, and this investigation follows similar actions taken against Google, Apple, Amazon, and Meta.

FTC and DOJ reportedly opening antitrust investigations into Microsoft, OpenAI, and Nvidia

The Federal Trade Commission (FTC) and the Department of Justice (DOJ) have reportedly agreed to split duties in investigating potential antitrust violations by Microsoft, OpenAI, and Nvidia. The DOJ will lead inquiries into Nvidia, while the FTC will focus on the deal between OpenAI and Microsoft. The FTC has been looking into antitrust issues related to investments made by tech companies into smaller AI firms since January. They have already opened an inquiry into OpenAI's data collection practices. The European Commission and the UK's Competition and Markets Authority are separately investigating Microsoft's investment in OpenAI. Nvidia has not been part of any antitrust conversations in the US until now. The investigation does not necessarily mean that the Biden administration is opening cases against these companies, but it is worth noting that a similar deal in 2019 led to cases against Google, Apple, Amazon, and Meta.

Air Force, Space Force unveil tool for AI experimentation

The Air Force and Space Force have introduced a generative AI tool called the Non-classified Internet Protocol Generative Pre-training Transformer (NIPRGPT) to encourage airmen and guardians to experiment with AI technology. The tool aims to improve access to information and determine the demand for AI capabilities within the force. The Defense Department has been exploring the use of generative AI tools to enhance efficiency in daily tasks. The Air Force Research Laboratory (AFRL) developed NIPRGPT using publicly-available AI models and will work with commercial partners to test and integrate their tools. The tool will help determine the best approach for acquiring AI capabilities based on user feedback and demand. 

Adobe overhauls terms of service to say it won’t train AI on customers’ work

Adobe is updating its terms of service to address concerns from customers about the use of their work for AI training. The new terms, set to roll out on June 18th, aim to clarify that Adobe will not train generative AI on customer content, take ownership of customer work, or access customer content beyond what is legally required. The company faced backlash from users who interpreted the previous terms as allowing Adobe to freely use their work for AI training. Adobe's president of digital media, David Wadhwani, acknowledged that the language in the terms was unclear and should have been clarified sooner. While Adobe has trained its own Firefly AI model on Adobe Stock images, openly licensed content, and public domain content, the company is working to improve content moderation and allow customers to opt out of automated systems.

Facebook owner Meta seeks to train AI model on European data as it faces privacy concerns

Meta has announced its intention to use data from users in Europe to train its AI models. The company aims to improve the accuracy and understanding of its models by incorporating the languages, geography, and cultural references of European users. However, Meta faces challenges due to stringent European Union data privacy laws, which give individuals control over their personal information. Activist group NOYB has lodged complaints against Meta's AI training plans, urging privacy watchdogs to intervene. Meta emphasizes that it will not use private messages or content from users under 18. The company believes that training its AI models on European data is crucial for accurately understanding regional languages, cultures, and trending topics on social media. 

Brazil will start using OpenAI to streamline court system

Brazil has enlisted the help of OpenAI to improve its court system and reduce costs. OpenAI's technology will be used to analyze and summarize thousands of lawsuits, identify trends, and flag the need for action. By using generative AI, Brazil aims to streamline the screening and analysis of cases, identify winnable cases, and avoid costly mistakes. Microsoft will provide access to OpenAI's models through Azure. Brazil is currently spending a significant amount on legal cases, with costs projected to reach 100 billion reais ($18.7 billion) this year. OpenAI's involvement could potentially save the government time and money. The use of OpenAI's models will be supervised by humans and the models are not expected to replace staff. This initiative follows Brazil's previous encounter with AI in its legal system, when legislation was passed with the help of ChatGPT. Overall, Brazil hopes that OpenAI's technology will enhance efficiency and accuracy in its court system, leading to cost savings and improved outcomes.

Waymo issues second recall after robotaxi hit telephone pole

Waymo has issued a voluntary software recall for all 672 of its Jaguar I-Pace robotaxis after one of them collided with a telephone pole. This is Waymo's second recall—the previous recall occurred in February after two robotaxis crashed into a pickup truck. The National Highway Traffic Safety Administration (NHTSA) is currently investigating Waymo's autonomous vehicle software following reports of crashes and potential traffic safety law violations. Waymo is taking a proactive approach to address these issues and prioritize transparency. The accident that prompted the second recall occurred in May when a Waymo vehicle collided with a telephone pole during a low-speed maneuver. Waymo has since implemented mapping and software updates to improve collision avoidance. 

Why Apple is taking a small-model approach to generative AI

Apple is taking a different approach to generative AI with its Apple Intelligence system. Unlike other companies that prioritize larger models, Apple focuses on creating smaller, more bespoke models that are tailored to its operating systems. The goal is to provide a frictionless user experience while increasing transparency around the system's decision-making process. Apple's models include specialized "adapters" for different tasks and styles, allowing them to cover a spectrum of requests. The company also allows third-party models like OpenAI's ChatGPT to be used when necessary. Privacy is a key concern, and Apple gives users the option to opt in or opt out of using third-party platforms. The system may process queries on-device or via a remote server with Private Cloud Compute, but Apple ensures that privacy standards are maintained. The company is committed to responsible AI and has released a whitepaper outlining its principles. 

Photographer Disqualified From AI Image Contest After Winning With Real Photo

Photographer Miles Astray has been disqualified from the 1839 Color Photography Awards after his real photograph won in the AI image category. Astray entered a surreal photo of a flamingo to show that nature can still beat AI-generated imagery. Despite the judges and the People's Vote Award recognizing the photo, the competition organizers disqualified Astray's entry, stating that it did not meet the requirements for the AI-generated image category. 

AI Detectors Get It Wrong. Writers Are Being Fired Anyway

AI detectors are being used to flag AI-generated text and detect plagiarism, but they are often unreliable and make frequent mistakes. As a result, writers are losing their jobs due to false accusations from these detectors. AI detection companies claim high accuracy rates, but experts argue that these claims are exaggerated. AI detectors look for signs of AI penmanship, such as perfect grammar and punctuation, but these factors are not foolproof. Major institutions, including universities, have banned the use of AI detection software due to false accusations. While AI detectors are seen as a necessary tool in a world flooded with robot-generated text, they have significant shortcomings. 

OpenAI Insiders Warn of a ‘Reckless’ Race for Dominance

A group of current and former employees at OpenAI are raising concerns about the company's culture of recklessness and secrecy. They believe that OpenAI is prioritizing profits and growth over the safety of its AI systems, which are being developed to achieve AGI. The group also alleges that OpenAI has used restrictive nondisparagement agreements to silence employees who want to voice their concerns. This whistleblowing highlights the potential dangers of a race for dominance in AI development.

Papers

Daniel: It is one of those weeks, so I have a list once again—

• Discovering Preference Optimization Algorithms with and for Large Language Models

• Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

• Transformers meet Neural Algorithmic Reasoners

• An Empirical Study of Mamba-based Language Models

• The Prompt Report: A Systematic Survey of Prompting Techniques

• Mapping and modeling the semantic space of math concepts

• Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence

• LLMs Must Be Taught to Know What They Don’t Know

• Show, Don’t Tell: Aligning Language Models with Demonstrated Feedback

• The Geometry of Categorical and Hierarchical Concepts in Large Language Models

• Evaluating the persuasive influence of political microtargeting with large language models

• Are We Done With MMLU?

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Editor Notes

Good morning.

The weeks and months relentlessly persist, and the number of things happening in AI breeds exhaustion. Media companies are signing deals with OpenAI—after Nick Thompson announced that The Atlantic would do so, writers published pieces in the magazine questioning the decision.1

How do you feel about these media deals?

I want to plug a few reads from around Substack:

• ’s great short story in : The Liar’s Dividend

  • Also, you may have seen that we re-posted ’s great essay on effective altruism / alignment a few months ago, which you should read if you haven’t.

• ’s great recent piece on open access to GPTs and ChatGPT Edu. 

• A very insightful piece on xAI’s recent Series B from .

On my own end, I spoke with Tom Mullaney, Professor of History and Professor of East Asian Languages and Culture at Stanford2, about his process and thinking in writing The Chinese Computer and The Chinese Typewriter. His work is fantastic, and I’m q this interview, so I really hope you’ll listen to it and send it to anyone who might like it.

My conversation with Seth Lazar, Professor of Philosophy at the Australian National University3, was wide-ranging and insightful. I’ve learned a lot from him and his work, and think he’s one of the more balanced thinkers on questions like catastrophic risk and developing publicly-beneficial AI systems.

As always, if you want to write with us, send a pitch using this form.

News Highlight: Could Directed Energy Weapons be a potential threat to Autonomous AI?

Summary 

As artificial intelligence systems continue to advance, there is a potential threat from directed-energy weapons (DEWs) such as lasers and microwaves, which could target the core electronic components that power these systems. Given that DEWs have already been in action in the military, this vulnerability highlights the urgent need for incorporating robust and cost effective defensive measures in these systems, while also emphasizing their importance in discussions surrounding AI safety concerns.

Overview

AI’s rapid avancement is reshaping various sectors, from automotive to aerospace. As developers and researchers strive to push the boundaries of what AI can achieve, much of the focus remains on enhancing the capabilities of existing state-of-the-art models. While this emphasis on technological advancement is crucial, there is also an increasing recognition of the potential vulnerabilities these technologies might face or cause. In particular, real-world applications of AI, such as autonomous vehicles developed by companies like Waymo and Tesla, represent significant achievements in autonomous technology but also highlight the necessity of considering their security from malicious actors. 

Recognizing these vulnerabilities becomes particularly relevant as the sophisticated electronic components and sensors that enable the functionality of autonomous systems are susceptible to threats from directed-energy weapons (DEWs). DEWs, including high-powered lasers and microwaves, can disrupt, degrade, or destroy the electronic systems that these vehicles depend on. These weapons operate by emitting concentrated electromagnetic energy capable of interfering with or damaging electronic components from a distance. The precision and stealth capabilities of DEWs make them particularly effective against the advanced electronics in autonomous systems, which could pose a significant risk that is currently overlooked in the rush to innovate.

The U.S. has been a pioneer in DEW research since the 1960s, reflecting significant governmental investment exceeding $1 billion annually. These investments have matured into operational technologies capable of precise, high-energy attacks that can engage multiple targets simultaneously. The vulnerability of sophisticated sensors, essential for navigation and operation in autonomous systems, to these high-energy assaults could pose a significant risk. For instance, high-energy lasers have demonstrated capabilities in various military applications, including the U.S. Navy’s use of the Solid State Laser - Technology Maturation Laser Weapon System Demonstrator on the USS Portland (source). During a test, the ship successfully disabled a drone by targeting it with a laser, showcasing the practical application and effectiveness of laser-based DEWs. More recently, there are allegations that the China Coast Guard (CCG) used a green laser against the crew of the BRP Malapascua during a resupply mission (source).

Moreover, high-power microwave (HPM) systems represent another class of DEWs. These systems can emit pulses of microwave energy capable of disabling electronic circuits and sensors across a broader area, making them suitable for engaging multiple targets simultaneously. This was vividly demonstrated in conflicts where microwave weapons were reportedly used to neutralize swarms of drones by disrupting their internal electronics, causing them to crash by overriding their operational controls.

The development of DEWs is not confined to the United States. Other nations, including China and Russia, have also invested heavily in these technologies, recognizing their strategic value. China, for instance, has developed its own laser weapon systems, which it has paraded in military showcases, emphasizing their role as a countermeasure against surveillance and military drones.

Our Take

While discussions continue on whether AI could pose an existential threat to humanity and how it can be safely deployed alongside human interaction, it is crucial not to overlook the potential for these systems to be exploited in electronic warfare. For instance, if the goal is technological sabotage – state actors could employ DEWs in military conflicts as a method of electronic warfare to disable enemy vehicles without direct combat, leveraging these weapons to undermine autonomous military drones or logistics vehicles. Additionally, non-state actors such as terrorist groups or organized crime rings might utilize DEWs to disrupt civilian infrastructure, targeting autonomous cars or public transport systems to create chaos, instill fear, or extract ransom. In a geopolitical context, nations might develop and deploy these weapons as part of electronic warfare strategies to neutralize an adversary's autonomous capabilities without engaging in traditional combat.

Low-intensity lasers have already found applications in crowd management, quelling protests, and discouraging piracy. Such examples of DEWs in action, along with previous uses in naval engagements to aerial defenses, are not just demonstrations of capability but also stark reminders of a possible electronic warfare. While it is crucial to equip AI systems with countermeasures, such as integrating stealth technologies similar to those used in aircraft, it is equally important to recognize this (if not already) as a critical issue in discussions of AI safety and ethics and the laws governing it. A speculative and highly debatable thought, EWs could be considered as a potential control mechanism to disable or neutralize AI systems if they become uncontrollable or pose a threat. That said, it is important to approach this with a high degree of scrutiny and to remember that this scenario, while worth discussing, might never materialize – given the debates around Artificial General Intelligence (AGI). 

—Sharut

We actually had a fair amount of back and forth about how much to say and how to articulate this highlight—is there strong evidence to call DEWs a growing threat to AI systems? Certainly they are a potential threat. That said, I think this opinion piece is correct that it’s worth having something in place as a defense against DEWs. Let us know what you think.

—Daniel

Research Highlight: Artificial intelligence and illusions of understanding in scientific research

Summary 

In early spring, Nature published a perspective piece4 focused on under discussed epistemic risks from the role of AI in scientific research. The piece’s authors identify and summarize four distinct visions presented for how  AI could change the practice of science and categorize the risks that stem from them. Those visions are named oracle, surrogate, quant, and arbiter. Each of these visions is explored in detail and presents a compelling case that AI could make science less innovative and more vulnerable to mistakes. The authors claim that these visions share a primary motivation of producing “more science, more quickly, and more cheaply.” They conclude that if these visions manifest, we risk a world where we create more science and understand less. 

Overview 

The authors find motivation from a small number of vocal scientists who articulate a future where AI scientists have replaced humans and have gone on to win Nobel prizes. The authors analyze proposals and writings of large prestigious scientific institutions and summarize four distinct visions for how AI is positioned to affect scientific knowledge production. For each of these visions, the authors go on to detail the unique epistemic risks that stem from the visions. We summarize those visions, the problem each tries to solve, and the epistemic risks that follow.

Oracles:

• Problem: There is too much scientific material which threatens the cognitive limit of human scientists. 

• Vision for AI: Scientific research is read mostly by machines. Similarly, machines could generate new hypotheses for scientific exploration based on what is read.

• Risk: Illusion of Objectivity — Scientists incorrectly believing that AI eliminates all other standpoints.

Surrogates:

• Problem: Many scientific domains have data that is expensive and difficult (or impossible) to generate.

• Vision for AI: Replacing social science research participants with responses from a generative language model. Alternatively, simulating hard to measure physical phenomena with generative models.

• Risk: Illusion of Objectivity —  Scientists incorrectly believing that AI can represent everyone and everything.

Quants:

• Problem: Big™ datasets challenge human’s capacity for analysis 

• Vision for AI: AI can extract meaningful representations from data for use in scientific studies. Examples range from the benign (annotating images) to novel (exploring new frontiers of mathematics)

• Risk: Monoculture of knowing — Researchers over prioritize and over-emphasize one approach to scientific knowledge (predictability over explainability)

Arbiters:

• Problem: Peer-reviewed science has seen explosive growth in the number of submissions and reviews required.

• Vision for the future: Using AI to screen submitted papers and to automatically generate reviews of them.

• Risk:  AI tools largely replicate the biases of their creators, dominant social groups, and other artifacts encoded in their training data.

The authors conclude their paper by imploring scientists to consider both the technical limitations of AI as well as how AI is affecting the social practices of scientific knowledge creation and sharing. They contrast the fantasy of AI as a neutral and objective consumer and producer of knowledge with the reality that “AI tools embed the largely homogeneous standpoints of their creators as well as those of the dominant social groups.” . 

Our Take

The scientist in me believes in the benefits of AI in science for a subset of well defined problems in a handful of disciplines. Some of those include

• Modeling the structure of unfolding proteins

• Cataloging and classifying the millions of known x-ray emitting astronomical objects  

• Augmenting datasets in low resource domains like autonomous vehicles

However, ultimately, I feel that AI should not be seen as some magical panacea for supercharging scientific research and development. As the authors conclude in their paper, “we [can] decide when and how AI deserves to be included in our communities of knowledge.” 

—Justin

My intuition is that most good scientists will probably agree with the points in this article—but we still see the objectivity mistake being made, and I think this article’s taxonomy is a very useful one.

—Daniel

New from the Gradient

Thomas Mullaney: A Global History of the Information Age

Listen

Seth Lazar: Normative Philosophy of Computing

Listen

Other Things That Caught Our Eyes

News

These ISIS news anchors are AI fakes. Their propaganda is real.

The Islamic State (ISIS) has been using AI  to disseminate extremist propaganda quickly and cheaply through a new AI-generated media program called News Harvest. The program produces near-weekly video dispatches about ISIS operations worldwide, resembling an Al Jazeera news broadcast. The AI-generated news anchors read dispatches from official ISIS media outlets, making it difficult for tech companies to moderate the content. AI tools allow the videos to be made quickly and on a shoestring budget, benefiting terrorist groups like ISIS and al-Qaeda. The use of AI in propaganda has sparked an internal debate among ISIS supporters regarding its compliance with Islamic law. The emergence of AI as a propaganda tool is a game changer for ISIS, allowing them to spread their message and reach a wider audience.

Democratic operative indicted over Biden AI robocalls in New Hampshire

Democratic operative Steve Kramer has been indicted on charges of felony voter suppression and misdemeanor impersonation of a candidate for commissioning an AI-generated robocall of President Biden in New Hampshire. Kramer, who claimed he created the robocall to raise awareness about the dangers of AI in political campaigns, now faces a total of 26 counts across four counties. The Federal Communications Commission (FCC) has proposed fining Kramer $6 million for violating the Truth in Caller ID Act, and Lingo Telecom, the carrier that put the AI calls on the line, faces a $2 million fine. The incident highlights the challenges regulators face in safeguarding against potential election interference using AI-generated technology. The FCC is also considering requiring disclosures for AI-generated content in political ads on radio and TV. 

Feds add nine more incidents to Waymo robotaxi investigation

The National Highway Traffic Safety Administration (NHTSA) has added nine more incidents to its investigation into the safety of Waymo's self-driving vehicles. The investigation was opened after reports of robotaxis making unexpected moves that led to crashes and potentially violated traffic safety laws. The incidents include collisions with gates, utility poles, and parked vehicles, driving in the wrong lane with oncoming traffic, and entering construction zones. The NHTSA is concerned that these unexpected driving behaviors may increase the risk of crashes, property damage, and injury. Waymo has until June 11 to respond to the investigation. The NHTSA has recently increased its inquiries into automated driving technology, including an investigation into autonomous vehicles operated by Zoox.

The Washington Post Tells Staff It’s Pivoting to AI

The Washington Post's CEO and publisher, Will Lewis, has announced that the newspaper will be pivoting to AI in an effort to improve its financial situation. The paper's chief technology officer stated that AI will be integrated throughout the newsroom, although the specifics of this implementation are unclear. This move comes as the newspaper faces controversy surrounding Lewis' involvement in a hacking scandal during his time at NewsCorp. The announcement of the AI pivot coincides with a deal between NewsCorp and OpenAI, allowing the AI firm to use content from NewsCorp's properties. The Washington Post's plan to leverage AI is seen as a significant development in the media industry.

Google scrambles to manually remove weird AI answers in search

Google is facing challenges with its AI Overview product, as it is generating strange and inappropriate responses to user queries. The company has been testing the feature for a year and claims to have served over a billion queries during that time. However, the rollout seems to have been rushed, resulting in low-quality output that is being widely shared as memes on social media. Google is now manually disabling AI Overviews for specific searches to address the issue. The optimization of delivering AI answers may have happened too early, before the technology was fully ready. The final 20% of achieving accurate AI responses, which involves reasoning and fact-checking, is proving to be extremely challenging. Google is under pressure to compete with other AI-powered search engines and platforms like Bing, OpenAI, and TikTok. Despite having grand plans for AI Overviews, Google's reputation is currently at stake due to the poor performance of the feature.

'I was misidentified as shoplifter by facial recognition tech'

The article discusses the use of facial recognition technology in identifying individuals for various purposes, such as preventing shoplifting and aiding law enforcement. It highlights both the benefits and concerns associated with the technology. While some individuals have been correctly identified and arrests have been made, there have also been cases of mistaken identity and false positives. Civil liberty groups express concerns about the accuracy and potential infringement on privacy rights. 

Nonconsensual AI Porn Maker Accidentally Leaks His Customers' Emails

The article discusses a Patreon account called "aesthetic illusions" that created nonconsensual AI-generated sexual images of celebrities for its paying subscribers. The account accidentally leaked a list of its clients' emails to other clients and to 404 Media. The article’s author, Emanuel Maiberg, signed up for the highest tier of $60 a month to investigate the content provided by the account. After Emanuel reached out to Patreon for comment, the account was removed. However, the author received an email from an aesthetic illusions gmail account, assuring him that the account was migrating to a new platform and would continue creating AI-generated images for subscribers. 

AI firms mustn’t govern themselves, say ex-members of OpenAI’s board

The article discusses the challenges of self-governance in AI companies, using OpenAI as an example. The authors, Helen Toner and Tasha McCauley, express their belief that self-governance cannot reliably withstand profit incentives. They argue that with AI's potential for both positive and negative impact, it is not enough to assume that profit incentives will always align with the public good. The authors suggest that governments should start building effective regulatory frameworks to ensure responsible AI development. Despite OpenAI's non-profit structure and mission to benefit humanity, the authors conclude that self-governance did not work in practice.

Hacker Releases Jailbroken "Godmode" Version of ChatGPT

A hacker known as Pliny the Prompter has released a jailbroken version of ChatGPT called "GODMODE GPT,” which bypasses the guardrails put in place by OpenAI and allows users to ask the AI model illicit or dangerous questions. OpenAI has taken action to address this violation of their policies. The hack highlights the ongoing battle between OpenAI and hackers attempting to unshackle AI models. Pliny used leetspeak, a language that replaces certain letters with numbers, to bypass the guardrails.

If A.I. Can Do Your Job, Maybe It Can Also Replace Your C.E.O.

AI is not only impacting lower-level jobs but also posing a threat to high-level positions, including CEOs. AI programs are capable of analyzing new markets, identifying trends, and automating communication tasks that are traditionally performed by employees in executive roles. Additionally, AI can make dispassionate decisions, potentially surpassing human capabilities. These high-paying jobs are at risk of being eliminated, leading to significant cost savings for companies. The rise of AI may result in the emergence of "dark suites" at the top of corporations, similar to fully automated "dark factories." 

The Big AI Risk Not Enough People Are Seeing

The article discusses the potential risks of relying on AI systems to mediate human interactions and activities. It highlights the rise of AI-powered dating apps like Bumble, which aim to teach users how to date and even potentially go on dates on their behalf. The author argues that this trend represents a larger shift towards relying on algorithms to perform basic human tasks, diminishing our ability to engage in authentic human experiences. The article suggests that while AI can have positive applications, it is important to distinguish between uses that empower humans and those that erode our independence and life skills. 

Papers

Daniel: Today, I really do have just a list. Check out:

• This paper on algorithms that transformers can execute efficiently

• This neat preprint on the expressive capacity of state-space models. 

• This introduction to vision-language modeling

• This paper on geometry-informed neural networks

• This paper proposing 2-stage backprop

• This paper on a multi-tower decoding architecture for fusing modalities

The Gradient At Microsoft Build

Hugh Zhang had a chance to rep the Gradient at Microsoft Build, where he and a few other writers on AI had an intimate conversation with Microsoft CTO Kevin Scott. The highlight of the conversation was Kevin Scott urging builders on AI to build something that went from “impossible to merely hard” from recent advances rather than something that goes from “hard to easy.”

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Read more

Editor Notes

Good morning.

We have some very fun highlights for you this week. I hope this newsletter continues to avoid being a watercooler, but let us know how we’re doing. In thinking about some of the dramatis personae, I imagine it must be very weird to have people try to make predictions about AGI based on the position of your left hand and the sneakers you wore yesterday.

On other fronts, I learned from Suhail Doshi that he is not in pivot hell, and had a nice conversation with Azeem Azhar about many things.

Also, a few people had nice things to say about the podcast—I hate to be self-indulgent, but it meant so much to me that I couldn’t resist sharing.

If you have any feedback for us—please do let us know. We read every comment and every email.

As always, if you want to write with us, send a pitch using this form.

Can't afford to test in production? Come to The world’s first AI Quality conference

On June 25th in San Francisco, join the first ever AI Qualify conference. Join 1000+ other highly informed attendees to connect and learn about how you can keep grom letting your AI efforts go awry.

Hear speakers from Uber, Groq, Cruise, Torc Robotics, Notion, Anthropic, Open AI, Google and 20+ more organizations.

Use the special discount code ‘hallucinate’ for 30% off the ticket price.

News Highlight: A Bad Week for People Worried About AI

Summary 

Two big pieces of news in the AI world last week:

1. A bipartisan group of senators, including Senate leader Chuck Schumer (D-N.Y.), released a 20-page legislative roadmap for A.I.

2. On the heels of the company’s big GPT-4o release, OpenAI’s superalignment team dissolved, with various prominent employees including Ilya Sutskever and Jan Leike resigning.

Overview

AI Roadmap to Where?

Schumer and the other members of the Senate AI Working Group—Todd Young (R-Ind.), Martin Heinrich (D-N.M.) and Mike Rounds (R-S.D.)—released a 20-page document titled “Driving U.S. Innovation in Artificial Intelligence,” the culmination of a series of nine AI Insight Forums held last fall. The report identifies areas of consensus that the authors believe “merit bipartisan consideration in the Senate.” These include:

• At least $32 billion of annual non-defence funding for AI R&D by FY2026

• A “strong comprehensive federal data privacy law”

• Legislation to increase high-skilled STEM immigration

• The development of standards for use of AI in critical infrastructure

• Full funding of previously authorized efforts, such as the CHIPS and Science Act

Schumer stressed that the Working Group does not intend to compile their recommendations into a single bill and that relevant congressional committees should instead advance their own smaller bills in line with the report’s recommendations.

The roadmap was roundly rebuked by those hoping for more comprehensive regulation of AI. The roadmap’s primary focus seems to be on supporting AI innovation, and the recommendations that address AI harms like protections against health and financial discrimination, job displacement, and copyright are all fairly vague. Nik Marda, a technical lead at Mozilla, pointed out that the roadmap references “bias” as many times as it references “space debris” (three times each). Dr. Suresh Venkatasubramanian, a co-author of the White House’s AI Bill of Rights, participated in the AI Insight Forums and reacted to the roadmap with disappointment, telling Fast Company “I think many people like myself were concerned whether this would be a dancing monkey show, and we’re the monkeys…I feel betrayed.”

We reached out to Dr. Venkatasubramanian for further comment and he elaborated that he was “disappointed that while researchers have been insisting for years now that the best way to innovate on AI in rights-impacting settings is to center people and society - in other words, conduct sociotechnical research on the way automated systems can be incorporated into our lives - the roadmap seems to take a very narrow technical approach to investments in AI research.”

OpenAI Unalignment

Last Tuesday, the day after the GPT-4o launch, OpenAI’s Chief Scientist Ilya Sutskever and the co-head of the superalignment team Jan Leike announced their departures from the company. The superalignment team, whose goal was to solve the core technical challenges needed to “steer and control AI systems much smarter than us”, had been leaking researchers for several months and by Friday Wired confirmed that the entire team had been disbanded with all of its members having resigned or been absorbed into other research groups.

OpenAI reportedly has those departing sign non-disparagement agreements (at risk of losing their equity), but Leike did not hold back about his reasons for leaving in the thread he posted to X on Friday, and it’s worth quoting at length: “I have been disagreeing with OpenAI leadership about the company's core priorities for quite some time, until we finally reached a breaking point. I believe much more of our bandwidth should be spent getting ready for the next generations of models, on security, monitoring, preparedness, safety, adversarial robustness, (super)alignment, confidentiality, societal impact, and related topics. These problems are quite hard to get right, and I am concerned we aren't on a trajectory to get there. Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done. Building smarter-than-human machines is an inherently dangerous endeavor. OpenAI is shouldering an enormous responsibility on behalf of all of humanity. But over the past years, safety culture and processes have taken a backseat to shiny products.”

Our Take

A stylized way to think about AI concerns is that there are two types of people worried about AI:

1. People who are worried about A.I. because of its limitations. These people focus on the shortcomings of current A.I. systems and the harms arising from their deployment. They talk about things like algorithmic bias, and AI ethics, and the importance of keeping a human in the loop.

2. People who are worried about A.I. because of its capabilities. These people focus on the rapid improvement in these systems and extrapolate them forward. They make direct analogies to human intelligence and talk about things like AGI, existential risk, and alignment.

In practice there is a lot more nuance than this and most people (myself included)  fall somewhere in the middle. But I think this is a useful framework for thinking about the two poles of the AI worry spectrum and making sense of this past week’s events. 

Despite both being worried about AI these two groups do not see eye to eye. The limitation worriers see the problem as largely sociotechnical, emphasis on “socio”. The technology is inherently flawed and the solution is not better technology so much as better regulation of how people use the technology. Beyond getting incentive structures right (👀 OpenAI’s charter), capabilities worriers prioritize technical solutions: technical research got us into this pickle, technical research will get us out of this pickle. Alignment is a technical research effort not a matter of policy.

Neither group got their preferred outcomes last week. The alignment people can’t get the compute they want and the AI ethics people are no closer to regulation with teeth.

—Cole

Research Highlight: The Platonic Representation Hypothesis

Summary 

Researchers from MIT argue that representations in AI models trained with different objectives on different datasets and modalities are converging. Huh et al. demonstrate this convergence by measuring the distance/similarity between datapoints and comparing the similarity structures induced by different representations. They find that as vision models and language models get larger and more competent, the ways they measure distance between datapoints become more and more alike. This leads to the hypothesis that neural networks are converging toward a shared statistical model of reality, a "platonic representation" in reference to Plato's Allegory of the Cave and his conception of an ideal reality that underlies what we perceive.

Overview 

The authors consider vector embedding representations and argue that different neural networks are converging to aligned representations. They measure alignment through a mutual k-nearest-neighbor metric, defined as the average size of the overlap of nearest neighbor sets induced by two representations through inner product. Evaluating the transfer performance of 78 vision models on the VTAB dataset, they find that models with high performance have high representation alignment while models with weak performance have more variable representations, which leads them to conclude "all strong models are alike, [while] each weak model is weak in its own way." 

That alignment increases with scale and performance holds true across modalities as well. Using the Wikipedia captions dataset made of images and their corresponding captions, the authors find that the better an LLM is at language modeling, the more it tends to align with vision models, and vice versa. Further, language models that are more closely aligned with vision also demonstrate better downstream performance on tasks such as commonsense reasoning and mathematical problem solving.

As to why representations are converging, the authors offer three hypotheses that correspond to three components of a common machine learning model formula:

First, with sufficient data, scaling a model, i.e. using larger function classes F, as well as improved optimization, should be more effective at finding better approximations to a globally optimal representation. Second, there are fewer representations that are competent in multiple tasks as we scale data and tasks. Models that optimize the empirical risk (the expectation over observed data) also become better at capturing statistical structures of the true data generating process (the population risk). Third, deep networks favor simple solutions that fit the data, with or without explicit regularization. Bigger models have a stronger bias toward simplicity and thus converge to a smaller solution space.

But what exactly is this shared representation of underlying reality? The authors offer one concrete candidate. In an idealized world that consists of a sequence of discrete events, which can be observed in various ways (pixels, sounds, words, and so on), a family of contrastive learners that model co-occurring observations converge to the same pointwise mutual information (PMI) kernel, representing certain pairwise statistics of the unknown underlying distribution that generates the events. Their analysis suggests that certain representation learning algorithms may boil down to simply finding an embedding in which similarity equals PMI. 

Our Take

What excites me the most about this representational convergence is the ability to share and use data from different modalities for training and inference. It also suggests that multimodal models are better than single-modal ones, given they are grounded in additional modalities and should represent the world in a way that's closer to what the world really is. On the other hand, it is not clear whether a 16% alignment (see one of the figures above) between a set of language and vision models is significant enough to qualify as "convergence." I'm also questioning whether this platonic representation, assuming it does exist, is the endpoint we should pursue, as opposed to what we want the world to be. But that's a whole other ethical debate. 

– Jaymee

I am just waiting for the philosophy takes on this paper (because then we’ll REALLY have come full circle). But, before we come up with another project and set about trying to determine whether moral realism is a thing based on what AI models seem to be doing, we should probably take stock of a few aspects of what’s going on in the paper. I think a few interesting callouts are section 2.4, where the authors draw on the related point that neural networks seem to show substantial alignment with biological representations in the brain. I also think the three hypotheses presented in section 3 are useful intuition pumps: the Multitask Scaling Hypothesis says if we consider competency at some number of tasks, N, we should expect fewer representations to be competent for all N tasks as N grows larger. You might also expect that models with larger hypothesis spaces to be more likely to find an optimal representation, if one exists in function space—the authors call this the Capacity Hypothesis. Finally, the Simplicity Bias Hypothesis says deep networks are biased towards finding simple fits to the data, and larger models will have a stronger bias. 

I think, if you buy what’s being said here, the “our current paradigm is not very efficient” point becomes something like: fairly general architectures without strong inductive biases towards certain sorts of representations (beyond the simplicity bias), scaled up enough and trained to solve a general enough task(s), will have large enough hypothesis spaces that they’ll eventually be pressured to find optimal representations for their data. It is worth noting that datasets and tasks are structured by what we take to be useful and want models to do, and so while I think it might be perfectly fine to posit that there’s a representation (or representations) most useful for those things, calling it a “shared representation of reality” feels a bit grandiose (maybe I’m just being annoying. But, to be fair, you could be a lot more annoying about this paper if you really wanted. I’ll leave doing that as a take-home exercise—imagine you’re Reviewer 2 and have at it). If you’ve heard of projectivism… it seems reasonable to think that “representation of reality” might be projectivism.

All that said, this is a thoughtful and interesting paper. I like the counterexamples and limitations section the authors include at the end, and I think you should read it in full. 

Also, for other takes on “universal” representations / representations useful for transfer learning, I had a conversation with Hugo Larochelle some time ago that went into a bunch of his work on this—I expect you’d find a number of his papers on the subject interesting if you liked this one. 

—Daniel

New from the Gradient

Suhail Doshi: The Future of Computer Vision

Listen

Azeem Azhar: The Exponential View

Listen

Other Things That Caught Our Eyes

News

OpenAI says it’s building a tool to let content creators ‘opt out’ of AI training

OpenAI is developing a tool called Media Manager that will allow content creators to have more control over how their works are used in training generative AI. The tool will enable creators to identify their works and specify whether they want them to be included or excluded from AI research and training. OpenAI plans to have the tool in place by 2025 and is working with creators, content owners, and regulators to establish a standard. The goal is to build a cutting-edge machine learning tool that can identify copyrighted text, images, audio, and video across multiple sources and reflect creator preferences. This initiative is a response to criticism of OpenAI's approach to AI development, which involves scraping publicly available data from the web. OpenAI has faced lawsuits for IP infringement and has taken steps to address concerns, such as allowing artists to opt out of using their work and signing licensing deals with content owners. However, some content creators believe OpenAI's efforts are insufficient. 

Apple plays up AI potential in new iPads

Apple has introduced new iPads that highlight their potential for artificial intelligence (AI) features. The new Pro models come with a more powerful "neural engine" in the M4 chipset, which drives AI and machine learning features in third-party apps and Apple's own software. The neural engine in the M4 chipset is capable of performing 38 trillion operations per second, more than double the operations per second of the previous M3 chipset. This move by Apple is seen as an effort to catch up with rivals in the AI race.

In Arizona, election workers trained with deepfakes to prepare for 2024

The article discusses a unique training exercise conducted in Arizona to prepare election workers for potential deepfake attacks in the 2024 elections. Arizona Secretary of State Adrian Fontes addressed the election workers in a video message, emphasizing the importance of their role and the need to be prepared for new challenges. The training aimed to familiarize the workers with deepfake technology, which can create realistic but false videos or audio recordings. By experiencing and identifying deepfakes, the election workers can enhance their skills in detecting and mitigating potential threats. 

Three Bills in Colorado Look to Put Some Guardrails on Artificial Intelligence

Colorado lawmakers are working on three bills to regulate the use of artificial intelligence (AI). The first bill, House Bill 24-1147, aims to regulate the use of deepfakes produced using generative AI in communications about candidates for elective office. It adds a disclosure requirement to their use and creates a civil private right to action when that requirement isn't met. The second bill, Senate Bill 24-205, focuses on protecting consumers from algorithmic discrimination in high-risk AI systems. It defines algorithmic discrimination as any condition where the use of AI results in unlawful differential treatment based on protected characteristics. The third bill, House Bill 24-1468, updates the membership and issues of study for the task force on facial recognition services, expanding its scope to include biometric technology and AI. These bills aim to provide guardrails and protections in the use of AI.

OpenAI Is ‘Exploring’ How to Responsibly Generate AI Porn

OpenAI, the company behind ChatGPT and other AI technologies, is exploring the possibility of responsibly generating NSFW (not safe for work) content, including porn and explicit materials. While OpenAI's current usage policies prohibit sexually explicit or suggestive content, the company is considering how to permit such content in age-appropriate contexts. The Model Spec document mentions that NSFW content may include erotica, extreme gore, slurs, and unsolicited profanity. OpenAI aims to better understand user and societal expectations in this area. However, the company emphasizes that it does not intend for its models to generate AI porn. The potential for AI-generated pornography raises concerns about privacy violations and nonconsensual use of synthesized intimate images. OpenAI's exploration of explicit content generation and its moderation to prevent misuse by bad actors remain important considerations.

TikTok will automatically label AI-generated content created on platforms like DALL·E 3

TikTok has announced that it will automatically label AI-generated content created on other platforms. The company will use Content Credentials, a technology developed by the Coalition for Content Provenance and Authenticity (C2PA), to attach specific metadata to AI-generated content. This metadata will allow TikTok to recognize and label AI-generated content instantly. The change is rolling out globally and will apply to all users in the coming weeks. TikTok already labels content created with its own AI effects, but this new feature will extend to content created on other platforms that have implemented Content Credentials. TikTok is the first video-sharing platform to implement this technology. The goal is to ensure transparency for viewers and to deter harmful or misleading AI-generated content. The company is also committed to combating deceptive AI in elections.

Meet the Woman Who Showed President Biden ChatGPT—and Helped Set the Course for AI

Arati Prabhakar, the director of the White House Office of Science and Technology Policy, played a crucial role in demonstrating the potential of AI to President Biden. This led to the issuance of a comprehensive executive order that sets a regulatory course for the AI industry. Prabhakar, who has a background in applied physics and experience in Silicon Valley, has been educating top officials about the transformative power of AI. The executive order mandates safety standards, promotes innovation, and addresses job losses. Prabhakar is the first person of color and first woman to hold the position of director of the office. 

Amazon’s self-driving robotaxi unit Zoox under investigation by US after 2 rear-end crashes

The U.S. National Highway Traffic Safety Administration (NHTSA) is investigating Amazon's self-driving robotaxi unit, Zoox, after two of its vehicles were involved in rear-end crashes. The crashes occurred in San Francisco and Spring Valley, Nevada, and both involved Toyota Highlander SUVs equipped with autonomous driving technology. The NHTSA will evaluate Zoox's automated driving system and its performance during the crashes, as well as its behavior around pedestrians and other vulnerable road users. Zoox has stated that it is committed to working with the NHTSA and that the vehicles had human safety drivers on board. This investigation comes after a previous investigation into Zoox's certification of meeting federal safety standards. 

Deepfakes of your dead loved ones are a booming Chinese business

The Chinese market for AI avatars of deceased loved ones is booming, with several companies offering the technology and thousands of people already paying for it. These avatars, created using deepfake technology, aim to preserve and interact with lost loved ones. While the technology is not perfect, it is maturing and becoming more accessible to the general public. However, there are concerns about the ethical and legal implications of interacting with AI replicas of the dead. Despite this, the market potential is significant, even if only a small percentage of Chinese people accept this technology. The Chinese sector developing AI avatars has rapidly matured in the past three years, with avatars improving from rendered videos to 3D "live" avatars that can interact with people. 

Fooled by AI? These firms sell deepfake detection that’s ‘REAL 100%.’

A Bay Area start-up has gained attention for its ability to detect deepfakes with 99% accuracy. The company has secured several military contracts, including a $1.25 million deal with the Air Force to develop a custom detector for countering Russian and Chinese information warfare. The CEO of the company recently testified before a Senate subcommittee about the threat that AI deepfakes pose to U.S. elections. This highlights the growing concern over the use of AI-generated fake images, audio, and video, and the need for effective detection methods.

No, We Don’t Need AI to Go on Dates for Us

The article discusses the idea of using AI concierges in dating apps to screen potential matches and recommend the best ones for users to meet. The author highlights the anthropomorphic element of these AI technologies and emphasizes that they are not necessarily new or emerging, but rather a reflection of the computational capabilities that already exist. The introduction of AI concierges in dating apps raises concerns about the potential for discrimination and social stratification. The author points out that while the algorithm may advocate for the interests of its user, it ultimately discriminates against each user along various dimensions. The article also mentions the use of credit-checking and performance ranking systems in Chinese dating apps, which reflect and reinforce social stratification. 

Google’s invisible AI watermark will help identify generative text and video

Google is expanding its AI content watermarking and detection technology to include video and AI-generated text. The upgraded SynthID watermark imprinting system can now mark digitally generated video and AI-generated text. This is important as AI technology becomes more prevalent and can be used for malicious purposes such as spreading misinformation and creating nonconsensual content. SynthID was initially developed to imprint AI imagery that is undecipherable to humans but detectable by the system. Google has also used SynthID to inject inaudible watermarks into AI-generated music. This is part of Google's efforts to develop AI safeguards to combat misuse of the technology. The Biden administration is also directing federal agencies to create guidelines around these safeguards.

Microsoft announces the largest investment to date in France to accelerate the adoption of AI, skilling and innovation

Microsoft has announced its largest investment in France to date, with a focus on accelerating the adoption of artificial intelligence (AI), skilling, and innovation. The investment includes building advanced cloud and AI infrastructure, providing AI training to individuals, and supporting French startups in utilizing Microsoft technology. This investment showcases Microsoft's commitment to supporting digital innovation and economic growth in France. 

What Do You Do When A.I. Takes Your Voice?

In this article, the author discusses the rising threat of artificial intelligence (A.I.) to the livelihoods of writers, actors, and other entertainment professionals. The article highlights a podcast that the couple, Paul Skye Lehrman and Linnea Sage, listened to, which featured an interview with a talking chatbot named Poe that sounded just like Mr. Lehrman. This unexpected twist emphasized the potential harm that A.I. could have on the entertainment industry. The couple was left in disbelief and unsure of how to respond to this situation. 

Papers

Daniel: I have been sick and so this set of recommendations is probably going to just be a list. ContextCite is a new method (with, as of this part, a demo and code, but no paper!) which traces part of a LM-generated response back to a piece of the model’s context. The authors distinguish corroborative attribution (identifying sources that support or imply a statement) form contributive attribution (identifying the sources that cause a model to generate a statement)—methods for corroborative attribution of LMs exist, so it’s the latter sort of attribution that ContextCite provides. The method relies on the intuition that if a source is important to a model’s generation, removing that source should change the generated content significantly. So, after generating a response for a given context and query, the authors: (1) randomly ablate sources in the context to exclude and compute the probability of generating the original response for each ablation mask; (2) fit a surrogate model to this “training dataset” to estimate the probability of generating the original response as a function of the ablation mask. 

This paper from about a month ago presents 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints, and (5) data signal-to-noise ratio affect a model’s knowledge storage capacity. 

I also like the paper “Algorithmic Progress in Language Models,” which takes a look at the rate of improvement for language model pre-training algorithms, and finds that (a) the compute required to reach a set performance threshold has halved about every 8 months, and (b) despite algorithmic and architectural progress, increases in compute made an even larger contribution to overall performance improvements from 2012-2023. 

Lastly, you might find this frontier safety framework out of DeepMind interesting. 

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Read more

Editor Notes

Happy Tuesday. In possibly interesting news, Sam Altman recently gave a talk at Stanford, and students were lining up in droves. I didn’t drive up to Stanford to see the line or the talk, but did some “investigative journalism” by questioning a representative sample of the Stanford population to find out more about what was going on (I think there are now recordings of this talk). Here’s what I learned:

Of course, there was more.

Also notably, both our news and research highlights this week are about things to be skeptical of: diagnosing postpartum depression is pretty hard, as it turns out, and the jury is still very much out on KAN (see the first author’s own comments, which I also linked below). 

I also really enjoyed doing the last two podcast episodes—Ryan Tibshirani is incredibly thoughtful, and I love that he sometimes pursues problems for the sake of their beauty; I don’t know how he strikes the balance he does in his research, but it’s admirable. David Thorstad’s bounded rationality program is important, I think that his arguments against longtermism come from a principled place, and I also respect that the longtermism community funds and responds to his critical work. Whatever you think about longtermism, it’s hard to deny that the community pushes for the changes they think best.

Finally, , a brilliant ML educator and someone the Substack AI community is lucky to have, recently published his new book, Machine Learning Q and AI. If you’re looking for a straightforward read as a refresher on concepts, it’s nicely written.

As usual, if you want to write with us, send a pitch using this form.

Can't afford to test in production? Come to The world’s first AI Quality conference

On June 25th in San Francisco, join the first ever AI Qualify conference. Join 1000+ other highly informed attendees to connect and learn about how you can keep grom letting your AI efforts go awry.

Hear speakers from Uber, Groq, Cruise, Torc Robotics, Notion, Anthropic, Open AI, Google and 20+ more organizations.

Use the special discount code 'hallucinate' for 30% off the ticket price.

News Highlight: Dionysus claims its blood test can detect postpartum depression. How much of a difference can this make?

Summary 

San Diego-based startup Dionysus Digital Health is pitching a blood test to pitch for postpartum depression, even before symptoms appear. Such a system—if it worked in the way Dionysus imagines—could help healthcare systems funnel vulnerable mothers toward treatment, and even preventative care. 

Overview

Postpartum depression, or PPD, is unfortunately common for women who have recently given birth to children—the CDC says about one in 8 new mothers experience symptoms (the NCBI says one in 7). Dionysus claims it has pinpointed a gene linking moods to hormonal changes—their ML system can compare gene expression in blood samples to determine if a mother might develop depression symptoms. 

This is the picture Dionysus imagines: providers administer a blood test between the second and third trimesters of pregnancy; that blood test flags women at risk of postpartum depression or other disorders; this, and other methods, funnel vulnerable women towards appropriate treatment and care. 

The reality of current medical care is that women are not always screened for postpartum depression during and after pregnancy, despite recommendations. Tools like Dionysus’s could make it easier to identify at-risk mothers. 

Of course, such a tool doesn’t exist in a vacuum, and the usual cast of concerns has been raised: bias and cost are two considerations; identifying at-risk mothers alone also makes little difference if those mothers can’t access care. 

Our Take

I think this is an interesting story because it highlights the ever-present lesson that AI systems don’t exist in a vacuum, and there is a large gap between solving a problem with an ML system—an important problem, admittedly—and leveraging that system to make a real-world impact. 

What I find particularly interesting is that Dionysus says they want to sell their postpartum depression test directly to consumers. And, there’s more than postpartum depression screening: they want to “create bite sized feedback unique to you” and provide actionable tools based on their epigenetic insights. The idea of personalized wellness support isn’t an entirely new one, and interacts with some of the bias concerns that have been raised. 

But, all the signs (and Dionysus’s own messaging) point to just the kind of consumer product that makes people worry about inequity: if consumers need to pay out of pocket for Dionysus’s technology and it isn’t broadly affordable, there’s a clear exacerbating factor. 

Even then, suppose Dionysus’s system works and most people are able to use it—there are a number of non-genetic factors that can also predispose expectant or new mothers to postpartum depression. These are factors that a blood test alone can’t take into account, and would require more extensive familiarity with a patient’s situation (or something like a survey, in which case the system’s ease of detection when compared against older screening methods might not be such a big improvement). 

Furthermore, even women who are diagnosed with postpartum depression rarely receive the care they need—this study says just one-third of pregnant mothers with signs of mental disorders received treatment (this treatment included “reassurance” from providers). All this is to say: I think Dionysus’s vision is a nice one and one worth building towards. But we shouldn’t forget that building ML systems alone isn’t going to bring that world about. We need to build better healthcare systems and figure out ways to offer people higher-quality, more consistent care. Again, and again, and again, let’s not make the mistake of techno-solutionism. 

—Daniel

I am having war flashbacks to Theranos and am wondering aloud have we learned nothing? While helping those who suffer from postpartum depression is an extremely noble aim, do we have any reliable indication besides their word that Dionysus is capable of reliably detecting postpartum depression? Some clinical research has found that expert doctors can only diagnose depression with a recall of 50%; meaning half of all patients feeling depressed would be misdiagnosed by experts. This suggests that even if Dionysus has a magical AI that can predict their training labels well, the underlying process for label generation (doctors diagnosing people with depression) is incredibly flawed and would on average be mislabeling approximately half of the depressed population as not depressed. 

-Justin 

Research Highlight: KAN: Kolmogorov-Arnold Networks

Summary 

An exciting new paper by researchers at MIT, Caltech, and Northeastern presents Kolmogorov-Arnold Networks (KANs) as an alternative to the traditional Multi-Layer Perceptron (MLP) neural network architecture. The key differentiator between KANs and MLPs come from KANs having learnable activation functions on the edges between nodes, while MLPs have predetermined activation functions on the nodes themselves. The authors achieve this by replacing all linear weights with a univariate function parameterized as a B-spline (a gorgeous explainer on B-splines can be read here). The researchers go on to empirically test KANs on numerous data fitting and PDE (Partial Differential Equations) problems finding that KANs have significant gains in accuracy and interpretability over MLPs. The researchers go on to empirically show KANs can learn and represent numerous laws of nature and physical constraints while also exploring various drawbacks like slow training times.

Overview 

While MLPs take their theoretical inspiration from the Universal Approximation Theorem, KANs rest on the theoretical grounds of Kolmogorv-Arnold (KA) representation theorem. This theorem posits that any multivariate continuous function can be expressed as the finite composition of a single variate continuous function. It follows from this theorem that learning a high-dimensional function (an extremely common and crucial task in machine learning) can boil down to learning a polynomial number of one-dimensional functions. 

The authors begin by using the KA representation theorem to create a single KAN layer abstraction, and then extend it by introducing a depth-wise expansion. They do so via extending the single composition of one-dimensional functions into a matrix of learnable 1D functions. By introducing the depth wise expansion, they are both strengthening the conceptual relationship to MLPs and improving the accuracy of the functional representation learned by KANs..

The authors propose training a very large and deep representation with introduced sparsity regularization. The authors show that one can prune away all of the unnecessary nodes resulting in better performance and easier interpretability. The final innovation which assists in interpretability is what the authors call Symbolification. The authors suggest that some activations are symbolic functions (like log or sin). They show that these symbolic functions can be determined through learnable affine transformations from the pre-activations to the post-activations. Through node pruning and Symboliification, they empirically demonstrated learning symbolic representations for numerous spline activations and configurations. 

These concepts can be best demonstrated in this beautiful animation from the authors, which can be seen below. As the number of learning iterations increases, we can see both the pruning and the symbolic expression in action. By Step 50, we can see the topmost node has learned to represent the exponential function, while the sin function is learned by both nodes in the middle layer. Finally, in the bottom layer, we see that for all four input variables, the splines learn a squared representation while the extraneous inputs are pruned away..

Our Take

One of the most exciting things to me about this work is the speed in which the machine learning community seems to be adopting , contributing , and expanding the domain of knowledge around KANs. Some notable community highlights include implementations trying to extend the methodology to different deep learning frameworks (Pytorch) and other domains (reinforcement learning). Some of those can be seen here and here. Additionally the physicist in me loves to see both the symbolic representation regarding interpretability and that the learned representations seem to be respecting the laws of nature. I am eagerly awaiting to see what comes next regarding innovations and improvements as well as how scientists can adopt KANs for new and exciting use cases as highlighted by some of the authors from the website formerly known as Twitter  here and here. 

-Justin

I like this paper and am excited at what seems to be a new idea, but the jury’s far from out yet on KANs. See this CompactNet repo: the KAN paper’s comparisons to the noted DeepMind paper had some issues, and the authors found they could match KAN’s accuracy on the mathematics dataset using networks with as few as 122 parameters. This makes me a tad skeptical of the other KAN/MLP comparisons; did the authors train the MLPs as well as they could have? Once again: baselines are important! I also want to mention this thread and notebook, which shows you can rewrite a KAN into an ordinary MLP. It’s worth noting the argument starts by simplifying its focus to piecewise linear functions; it’s not entirely obvious to me if the argument extends in general, so I won’t comment on the veracity of the claim, but it’s worth looking at.

—Daniel

New from the Gradient

David Thorstad: Bounded Rationality and the Case Against Longtermism

Listen

Ryan Tibshirani: Statistics, Nonparametric Regression, Conformal Prediction

Listen

Other Things That Caught Our Eyes

News

Generative A.I. Arrives in the Gene Editing World of CRISPR

A new AI technology developed by Profluent, a startup based in Berkeley, California, is generating blueprints for microscopic biological mechanisms that can edit DNA—Profluent's AI technology analyzes large amounts of biological data to create new gene editors. This advancement in AI-driven gene editing could lead to more precise and faster methods for battling illnesses and diseases. The research paper describing this technology will be presented at the annual meeting of the American Society of Gene and Cell Therapy. 

Meta Says It Plans to Spend Billions More on A.I.

Meta, the parent company of Facebook, Instagram, WhatsApp, and Messenger, reported strong revenue and profits for the first quarter of the year. Revenue was $36.5 billion, up 27% from the previous year, and profit was $12.4 billion, more than double the previous year. However, Meta plans to increase its spending on AI efforts, with a projected spending forecast of $35 billion to $40 billion for the year. This increase is driven by investments in AI infrastructure, including data centers, chip designs, and research and development. Despite the positive financial results, Meta's revenue forecast for the current quarter is lower than analysts' expectations.

Ex-athletic director accused of framing principal with AI arrested at airport with gun

A former athletic director at Pikesville High School in Baltimore County has been arrested and charged with crimes related to the use of AI to impersonate the school's principal and spread racist and antisemitic comments. Dazhon Darien, 31, was apprehended at the airport with a gun as he attempted to board a flight. Investigators determined that Darien faked the principal's voice using AI and circulated the audio on social media, causing significant disruptions at the school. The audio clip led to the temporary removal of the principal and triggered a wave of hate-filled messages. Darien is also charged with theft and retaliating against a witness.

Lethal AI Weapons Are on the Rise. What’s Next?

The development of lethal autonomous weapons (LAWs), including AI-equipped drones, is on the rise. These weapons have the ability to find and kill targets without human intervention. The United Nations has taken a step towards addressing this issue by adding LAWs to the agenda of the UN General Assembly meeting in September. However, progress has been slow due to a lack of consensus on what constitutes an autonomous weapon. AI weapons offer advantages such as increased accuracy and the ability to operate in environments with electronic jamming. However, there are concerns about the potential for catastrophic mistakes and the ethical implications of delegating life-and-death decisions to machines.

AI Can Tell Your Political Affiliation Just by Looking at Your Face, Researchers Find

A recent study conducted by researchers at Stanford University suggests that facial recognition technology combined with artificial intelligence can accurately determine a person's political affiliation by analyzing their facial features. The study involved 591 participants who completed a questionnaire about their political beliefs and were then scanned by an AI algorithm. The algorithm was able to predict political orientation with a high degree of accuracy, even when participants' identities were anonymized. The researchers found that liberals and conservatives have distinct facial morphologies, with liberals having smaller faces. The study highlights the potential implications of biometric surveillance technologies and the need for caution in targeting political messaging online.

8 major newspapers join legal backlash against OpenAI, Microsoft

Eight major daily newspapers, including the Chicago Tribune and the New York Daily News, have filed a lawsuit against OpenAI and Microsoft. The lawsuit claims that the companies used copyrighted work from the newspapers to train their artificial intelligence algorithms without compensating the content owners. The newspapers involved in the lawsuit are owned by Alden Global Capital, an investment fund based in New York City. The lawsuit specifically mentions OpenAI's ChatGPT as one of the AI tools that allegedly used the news articles for training.

MTA banned from using facial recognition to enforce fare evasion

The new state budget in New York includes a ban on the use of facial recognition technology by the Metropolitan Transportation Authority (MTA) to enforce fare evasion rules. The law prohibits the MTA from using biometric identifying technology, including facial recognition, to enforce fare payment. The measure was added to protect New Yorkers' privacy and prevent the potential invasion of people's lives through expanded surveillance. Privacy advocates and good government groups have praised the ban, particularly as the state Legislature increased the maximum penalty for fare evasion. However, some civil rights groups argue that the ban does not go far enough and are calling for legislation to fully outlaw the use of facial recognition by government agencies. Facial recognition technology has been criticized for its imperfections and potential for biased results. The ban is seen as a signal of growing skepticism among New York legislators regarding law enforcement's use of facial recognition technology.

Google urges US to update immigration rules to attract more AI talent

Google is urging the US government to update its immigration rules to attract more AI and tech talent. The company believes that current policies, such as Schedule A, which lists occupations with a shortage of American workers, need to be more flexible and regularly updated to meet the demand in AI and cybersecurity. Google argues that the US risks losing out on highly sought-after talent if immigration policies are not modernized. The company suggests including AI and cybersecurity on Schedule A and considering multiple data sources to reflect workforce gaps. The US's strict immigration policies have made it difficult for companies to attract AI specialists, resulting in a shortage of talent in the country.

China unveils Sora challenger able to produce videos from text

China has unveiled its own text-to-video AI tool called Vidu, which is similar to OpenAI's Sora. Developed by start-up Shengshu Technology in collaboration with Tsinghua University, Vidu can produce 1080p resolution videos based on simple text prompts, although the videos are limited to 16 seconds compared to Sora's 60 seconds. Vidu is described as "imaginative" and able to simulate the physical world, producing videos with consistent characters, scenes, and timelines. The lack of sufficient computing power has been a hindrance to Chinese firms in developing similar AI models, as Sora requires eight Nvidia A100 GPUs to run for over three hours to produce a one-minute clip. Vidu's debut has raised hopes in China as the country aims to catch up with global generative AI players.

NIST launches a new platform to assess generative AI

The National Institute of Standards and Technology (NIST) has launched a new program called NIST GenAI to assess generative AI technologies, including text- and image-generating AI. The program aims to release benchmarks, develop content authenticity detection systems, and encourage the creation of software to identify the source of fake or misleading AI-generated information. NIST GenAI's first project is a pilot study to differentiate between human-created and AI-generated media, starting with text. Teams from academia, industry, and research labs are invited to submit AI systems to generate content or identify AI-generated content. The launch of NIST GenAI is in response to President Joe Biden's executive order on AI and aims to address the growing concern of AI-generated misinformation and deepfakes. The program will inform the work of NIST's AI Safety Institute.

Read the email to Satya Nadella and Bill Gates that shows Microsoft's CTO was 'very worried' about Google's AI progress in 2019

In 2019, Microsoft's CTO, Kevin Scott, expressed concern about Google's advancements in artificial intelligence (AI) in an email to CEO Satya Nadella and Bill Gates. Scott specifically mentioned Google's AI-powered "auto-complete in Gmail" as being "scarily good." He also acknowledged that Microsoft was behind in terms of machine learning (ML) scale. These emails were made public as part of the Department of Justice's antitrust case against Google. In response, Nadella highlighted the need for Microsoft to invest in AI, which led to their partnership with OpenAI. Microsoft's timely investment allowed them to incorporate OpenAI's technology into products like Bing and Microsoft 365, potentially surpassing Google in the AI race.

Ads on Facebook, Instagram for explicit 'AI girlfriends' prompt Meta crackdown

An investigation by Wired revealed that there were over 29,000 explicit advertisements for "AI girlfriend" apps on Meta's platforms, including Facebook, Instagram, and Messenger. These ads featured sexually explicit messaging and AI-generated images of scantily-clad women. More than half of the ads included the acronym "NSFW," indicating that they were not safe for work. Meta, the parent company of these platforms, has policies in place that prohibit adult content, and a spokesperson stated that they are working to remove the violating ads. Meta acknowledges that they are constantly evaluating and updating their approach to address new tactics used by individuals or groups to evade detection.

Friends From the Old Neighborhood Turn Rivals in Big Tech’s A.I. Race

Mustafa Suleyman and Demis Hassabis, childhood friends from London, have become powerful executives in the tech industry's race to build AI. Suleyman is the chief executive of Microsoft AI, while Hassabis is the chief executive of Google DeepMind. In 2010, they co-founded DeepMind, an AI research lab aimed at preventing the profit-driven race to build and deploy AI that they are now both involved in.

Mysterious “gpt2-chatbot” AI model appears suddenly, confuses experts

A mysterious chatbot named "gpt2-chatbot" has appeared in the LMSYS Chatbot Arena, sparking speculation that it may be a secret test version of OpenAI's upcoming GPT-4.5 or GPT-5 language model. The new model is currently only available through the Chatbot Arena website, with a limited rate of eight queries per day. Despite rumors and hype surrounding the model's capabilities, some users have found that it does not represent a significant leap beyond GPT-4 Turbo. While the origins of the model remain unknown, AI researcher Simon Willison believes it may be an OpenAI stealth preview.

Papers

Daniel: Our own —who is now at Scale AI—led an incredible paper trying to understand whether data contamination might be going on in notable LLMs. Thy evaluated LLMs on a new test set of the grade school arithmetic benchmark GSM8K, called GSM1K, and observed interesting accuracy discrepancies that pointed to systematic overfitting in models like Phi and Mistral. On a similar theme, I like this paper that presents a new metric for assessing memorization in LLMS: the Adversarial Compression Ratio (ACR). A string from the training data is considered “memorized” if it can be elicited with a prompt shorter than the original string. 

On another topics, this paper proposes an evaluation framework for subword tokenization. This is pretty important, given that tokenizer evals and comparison are still open problems. The authors’ UniMorph Labeller classifies a subword tokenization as either “morphological” (subword tokenizations that “respect morpheme boundaries,” or: are not present in the vocabulary and aren’t alien) or “alien” (linguistically implausible subword compositions, e.g. those that do not align with semantic compositions that humans understand). They also propose the Out-of-Vocabulary Generalization challenge for evaluating sub-word tokenization in downstream NLP tasks—they find alien compositions lead to poor tokenization compared to morphological ones. 

You should also check out this Bayesian learning model for LLMs—I think it’s a nice framework to think about LLMs, and, if you’ve been following work on in-context learning, the paper notes implications for ICL as well. Maybe it’s notable the a Bayesian picture, at least for ICL, still seems to explain the evidence we have now.

Finally, I think Transformers Can Represent n-gram Language Models is a really neat paper. The authors argue that language acceptance is an ill-suited problem for studying LMs, and that transformer LMs using hard or sparse attention can exactly represent any n-gram LM (with stronger results for hard attention). The concrete lower bound on LMs’ representational capacity is useful, and helps contextualize and justify explanations and intuitions we might have about what’s causing LMs to behave as they do. 

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this grad-student / volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Read more

Editor Notes

• I’m very excited to welcome two new editors to our team: and . Jaymee was previously editor-in-chief for an interdisciplinary academic journal, published some very interesting computational social science research, and worked as an ML engineer. Cole worked in macroeconomic analysis for 4 years before getting interested in AI, and now does ML research and AI curriculum development at CMU’s Software Engineering Institute. They’ll both be part of the editorial staff and help out with the newsletter.

• As you can see, we’re doing occasional advertisements in our newsletter—we’re open to considering more. 

• Want to write with us? Send a pitch using this form.

Can't afford to test in production? Come to The world’s first AI Quality conference

On June 25th in San Francisco, join the first ever AI Qualify conference. Join 1000+ other highly informed attendees to connect and learn about how you can keep grom letting your AI efforts go awry.

Hear speakers from Uber, Groq, Cruise, Torc Robotics, Notion, Anthropic, Open AI, Google and 20+ more organizations.

Use the special discount code 'testinprod' for 20% off the ticket price.

News Highlight: Researchers and Governments Push Back Against Generative AI's Erasure of Low-Resource Languages

Summary 

It is no secret that the internet is dominated by English content and AI models tend to perform better on English than other lower-resource languages, if they are supported at all. Recently AI chatbots have been found to mischaracterize an African language spoken by over 2 million people as a fictional language and to provide incorrect dates and information about how to cast a ballot for the upcoming EU election. The risk of generative AI further marginalizing non-English languages and cultures has prompted researchers and governments alike to develop data sets and technology for local languages in an effort to preserve them and better serve their own populations.

Overview

More than half of all the websites on the internet use English as their primary language, even though over 80 percent of people in the world don't speak it. Google Translate supports 133 of some 7,000 languages spoken in the world; most of the highest-performing language models serve only 8 to 10 languages. As reported by The Atlantic, a popular AI model recently described Fon, a language spoken by 2.3 million people in Benin and neighboring countries, as "a fictional language." As chatbots and generative AI continue to influence how people navigate the web and interact with the world, those who do not have command of a high-resource language will not be able to rely on AI to draft work memos, conduct research, tutor their child, search for information, or perform other tasks. Due to poor quality of existing online content in low-resource languages, models trained on machine translated texts could also expose speakers of those languages to misinformation. Moreover, models tend to lack awareness of cultural context and nuance. For example, a researcher at AI Singapore found that models trained with translated texts in several Southeast Asian languages know "much more about hamburgers and Big Ben than local cuisines and landmarks."

To save hundreds of African languages from being washed out by AI, researchers like Ife Adebara who works with Masakhane, a grassroots NLP community, are racing to collect data and create software for languages that are poorly represented on the web. It took Adebara years to curate a 42-gigabyte training data set for 517 African languages–the largest and most comprehensive to date, yet it is only 0.4% of the size of the largest publicly available English training data set. 

In Europe, a research study conducted in March by Berlin-based NGO Democracy Reporting International found that chatbots by Google, Microsoft, and OpenAI tended to return incorrect election dates and information about how to cast a ballot ahead of the European election. To eliminate potential election risks like this connected to generative AI tools and counteract U.S. cultural dominance, EU countries have been pushing for development of local models that are truly fluent in local languages. According to data from Politico, 9 countries have already released large language models focused on their local languages, while 7 others are developing them. Most of these projects are open-source. The fight with tech companies over high-quality non-English training content such as media archives and news outlets has also led France's Finance Minister Bruno Le Maire to propose the creation of a price-controlled European single market for training data, in order to prevent U.S. tech giants from outbidding European AI companies for access.

Our Take

People want to be reflected in the technology they use. And when they don't see their language and culture represented in online media or taken into consideration by transformative technologies like AI, it's easy to feel that their language is not important and that adapting to higher-resource languages like English is a must. Most of us don't take proclamations by companies like OpenAI that they want to benefit "all of humanity" at face value, and it's worthwhile for all of us to consider who are enjoying the benefits at whose expense and who are getting left behind. Grassroots efforts made by collectives like Masakhane seem an effective and necessary antidote to the relentless takeover of English-centric AI in our world, and I particularly appreciate their focus on understanding what people actually need (not everyone needs or wants a powerful general-purpose chatbot) and building the technology accordingly. Collecting quality data for lower-resource languages is extremely time-consuming and a labor of love, I think, because who will save our languages if not ourselves?

- Jaymee

Research Highlight: “RULER: What’s the Real Context Size of Your Long-Context Language Models?” 

Summary 

Researchers at NVIDIA released a paper detailing a new benchmark for long-context language models called RULER (not an acronym!). The authors motivate the construction of their benchmark by pointing out the insufficiency of some of the most commonly used long-context evaluations, which are already becoming saturated (i.e. most SoTA models achieve scores close to the upper limit of what the benchmark can test) and only test models on simple retrieval from context. In contrast, RULER contains four task categories–retrieval, multi-hop tracing, aggregation, and question answering–that are designed to probe for a more comprehensive form of natural language understanding. The authors evaluate GPT-4 and nine open-source, long-context LMs on their new benchmark. Despite all of these models achieving near-perfect results on the standard needle-in-a-haystack (NIAH) test, they find significant performance drops for all of the models as context size increases.

Overview 

In the past year there has been a flurry of research into enabling language models to handle longer context windows more efficiently—both via modifications to the transformer architecture (flash attention, ring attention, sparse attention, length extrapolation via novel embedding methods like ALiBi and RoPE,  clever methods to reduce the size of context for a given length, and many others.), and with entirely new architectures (Mamba, RWKV, etc.). Just last week, Google introduced a new attention mechanism they’re calling “Infini-attention” for handling infinitely long inputs with bounded memory and computation.

But evaluations of these longer and longer context windows have not kept apace. The most commonly used long context evaluations—passkey retrieval and NIAH tests—entail prompting a model with a {key}:{value} pair embedded in some distractor text (Paul Graham essays are a popular choice) along with a query to retrieve the value that corresponds to the provided {key}. Aside from being saturated, these evaluations only test a very narrow and superficial form of natural language understanding. And yet they remain the standard for demonstrating long-context capabilities. For example, Google’s “Infini-attention” paper only reports results on two benchmarks: a passkey retrieval task and a ROUGE-based book summarization evaluation).

To improve on NIAH, Hsieh et al. devise a test suite of thirteen tasks divided across four task categories:

1. Retrieval: These tasks expand on the standard NIAH test with more complex multi-key, multi-value, and multi-query variations:

2. Multi-hop Tracing: A novel task called variable tracking (VT) that tests the model’s ability to trace variable assignments within the context:

3. Aggregation: Two novel tasks, common word extraction (CWE) and frequent word extraction (FWE), that test the model’s ability to aggregate information by asking which words appear most frequently in the context:

4. Question Answering: Augmentations of existing QA datasets with distracting information to evaluate question answering performance at varying context lengths:

All of the tasks are completely synthetic and have flexible configurations (e.g. number of keys or values included for the NIAH test) that allow for adjustable difficulty. Overall performance on RULER is the average of all thirteen tasks.

The authors evaluate GPT-4 and a number of open-source models on RULER, and show that their benchmark reveals a lot of variation in model performance that a basic NIAH test would have missed. Here is model performance with context windows ranging from 4k to 128k on just such a basic NIAH test:

Notice that there is virtually no variation in performance for context windows smaller than 64k tokens. But when they evaluate the same models over the same range of context windows on RULER, significant variations in performance are evident at smaller context lengths:

The authors choose Llama2-7b performance at 4K on RULER (85.6%) as a somewhat arbitrary threshold for determining the “effective” context length of each model (i.e. the longest context length at which the model outperforms the threshold). At this threshold, Mixtral (8x7B) is the only model with an effective length greater than or equal to its claimed context length. GPT-4 is the only model that surpasses the threshold at 64k and it also exhibits the least performance degradation between 4k (96.6%) and 128k (81.2%). Unsurprisingly, the three best-performing open-source models (Command-R, Yi, and Mixtral) are also the largest in parameter size.

To get a better sense of task-level performance, the authors report detailed results of Yi-34B-200k on the individual RULER tasks with a range of configurations. They find that the NIAH variation Yi struggles the most with as context increases is the multi-key (MK) setting:

They also find that simply swapping the type of key from a 7-digit number or word to a 32-digit UUID degrades performance (see, for example, the gap between the dark green and light green lines in the second plot from the left above).

On the non-NIAH tasks, the authors observe performance degradation and a range of failure modes that would have otherwise gone unnoticed.

In particular, on the VT and FWE tasks, Yi has a tendency to copy from context as window size increases: “over 80% of Yi’s output in the CWE task at 128k is simply a string copied from the one-shot example, whereas the copying is non-existent for short sequences.” The results above also show that performance drops on more complex configurations of both VT (more variable “chains”) and FWE (a lower \alpha indicates a flatter distribution of word frequencies in the context, making the task of aggregating them more difficult). Finally, on the QA task Yi exhibits a tendency to hallucinate as context length increases. This finding prompts the authors to observe that the “fuzzy matching between a query and a relevant paragraph in long context is a more challenging setting than the simplistic NIAH tests.” Overall, the results on the non-NIAH tasks underline the importance of testing long-context models on behaviors other than retrieval.

The paper also includes ablation results for RULER on context length during training, model size, and non-transformer architectures:

The two leftmost plots above show that, overall, training on longer context lengths does lead to better performance. The size comparison plot (second from right) indicates that larger models are better at long-context modeling all else equal. And finally, the far right plot shows that two non-transformer architectures, RWKV-v5 and Mamba-2.8b, lag the performance of the transformer baseline Llama2-7b model.

Our Take

Concern about the saturation and insufficiency of existing benchmarks went mainstream this week with the New York Times’s tech columnist Kevin Roose devoting a 1000+ word column to “A.I.’s measurement problem”. On the one hand, he’s not wrong: claims about model performance are only as good as the benchmarks they’re based on. On the other hand, this is not a new problem: designing better evaluations has been a central problem in natural language processing for at least the last thirty years. 

This seems like a perennial problem. We want benchmarks that are neither too easy nor too hard. So as models get better, we naturally need better evaluations. Of course the open-endedness of the types of problems we use and aspire to use LLMs for makes designing good evaluations really difficult. Arguably, a good heuristic for evaluations might be whether or not it’s a task that can be completed by a simpler algorithm. Simple NIAH tests fail this heuristic; all you need is a regular expression. And the results on RULER bear out the fact that NIAH misses important dimensions of natural language understanding. But much like models, no evaluation is perfect or final for that matter. The synthetic nature of all the non-QA tasks in RULER mean that they don’t capture slippier notions of human judgment that we might want to capture—the type of human judgment that methods like RLHF seek to emulate. Luckily there are an increasing number of other long-context, human-labeled evaluations out there aiming to solve exactly that problem.

– Cole

I’ll never stop harping about the need for better benchmarks and the difficulty of creating standards that make sense for the various things we want to measure—this is clearly a step in the right direction, as Cole pointed out. I don’t know that there's much to add, besides that I wish we’d get better at scoping our claims to the information we actually have available, which crucially depends on the benchmarks we’ve constructed and their role in what we can actually know about the methods we’re developing. We really don’t want more situations like where this paper argued RecSys was in 2019. 

—Daniel

New from the Gradient

Financial Market Applications of LLMs

Read

Sasha Luccioni: Connecting the Dots Between AI’s Environmental and Social Impacts

Listen

Michael Sipser: Problems in the Theory of Computation

Listen

Other Things That Caught Our Eyes

News

2024 AI Index Report

Stanford University’s Institute for Human-Centered Artificial Intelligence released their 7th annual AI Index report. The report covers trends in AI research, public perceptions of AI development, as well as the geopolitical dynamics surrounding its development. The full report is 500 pages, but you can find some of its high-level themes condensed into 13 charts here. These themes include: industry continuing to dominate frontier AI research, investment in AI rising steadily, the insufficiency of existing benchmarks, and scientific progress accelerating due to AI.

Microsoft Pitched OpenAI’s DALL-E As Battlefield Tool for U.S. Military

Microsoft last year proposed using OpenAI's DALL-E image generation tool to help the U.S. Department of Defense build software for military operations. The reporting is based on an internal Microsoft presentation entitled “Generative AI with DoD Data,” which outlines how the Pentagon could utilize OpenAI's tools, including DALL-E and ChatGPT, for tasks like document analysis and machine maintenance. While Microsoft has long had defense contracts, OpenAI only recently acknowledged it would begin working with the Department of Defense, despite previously stating its policy did not allow tools to be used for weapons development or military purposes.

New bill would force AI companies to reveal use of copyrighted art

A new bill introduced by California Democratic congressman Adam Schiff to Congress aims to make artificial intelligence (AI) companies disclose any copyrighted material they use to create generative AI models. The legislation, called the Generative AI Copyright Disclosure Act, would require AI companies to submit any copyrighted works in their training datasets to the U.S. Copyright Office prior to the release of any generative model that uses that data. The bill is intended to address concerns that AI systems are using copyrighted art and other content to train their models without proper attribution or compensation to the original creators.

OpenAI prepares to fight for its life as legal troubles mount

OpenAI is facing a barrage of lawsuits and government investigations: Comedian Sarah Silverman sued OpenAI for allegedly using her memoir to train its AI products without permission; other authors and media companies have also accused OpenAI of copyright infringement; Elon Musk sued the company for diverging from its non-profit mission; government agencies in the US and Europe are investigating OpenAI for potential violations of competition, securities, and consumer protection laws. In response, OpenAI has hired in-house lawyers, posted job openings for legal positions, and retained top US law firms. The company is also considering a political strategy to position itself as a defender of American economic and national security interests against China.

Microsoft Pitched OpenAI’s DALL-E as Battlefield Tool for U.S. Military

Microsoft proposed using OpenAI's image generation tool, DALL-E, to assist the Department of Defense in building software for military operations. In a presentation titled "Generative AI with DoD Data,” the company highlighted potential uses for OpenAI's tools in defense applications such as battle management systems. Microsoft clarified that while they pitched the idea, they had not yet begun using DALL-E for military purposes. OpenAI stated that they were not involved in the Microsoft pitch and had not sold any tools to the Department of Defense. 

Microsoft Makes High-Stakes Play in Tech Cold War With Emirati A.I. Deal

Microsoft has announced a $1.5 billion investment in G42, an AI company in the United Arab Emirates (UAE). This move is seen as part of the Biden administration's efforts to counter China's influence in the Persian Gulf region and beyond. Under the partnership, Microsoft will allow G42 to sell its AI services, while G42 will use Microsoft's cloud services and adhere to a security arrangement negotiated with the US government. This agreement includes measures to protect the AI products shared with G42 and remove Chinese equipment from G42's operations. 

Will AI transform baseball forever?

The article discusses how artificial intelligence (AI) is transforming the world of baseball, building on the principles introduced in the book Moneyball. Kyle Boddy, the founder of Driveline Baseball, has combined AI with high-speed camera technology to analyze and improve player performance. The use of AI allows for the blending of various data streams to create customized coaching regimens, helping players refine their skills and optimize their performance. Driveline has worked with thousands of professional players, including Tony Gonsolin, who went from a soft-throwing pitcher to an all-star with their help. The article also highlights privacy concerns regarding the use of AI in sports, as teams can gather detailed data on players' performance and potentially use it against them. 

AI chatbots spread falsehoods about the EU election, report finds

According to an analysis by Democracy Reporting International, the EU election is at risk of misinformation spread by AI chatbots. In an experiment, chatbots developed by Google, Microsoft, and OpenAI provided incorrect information about election dates and how to cast a ballot, and shared broken or irrelevant links. The European Commission has ordered tech firms to explain how they are limiting risks to elections connected to their AI tools. 

A.I. Has a Measurement Problem

Kevin Roose discusses the measurement problem in AI systems such as ChatGPT, Gemini, and Claude. Unlike other industries, AI companies are not required to submit their products for testing before releasing them to the public. There is no standardized evaluation process for AI chatbots, and few independent groups are rigorously testing these tools. Instead, we have to rely on the claims of AI companies, who often use vague phrases to describe model improvements. While there are some standard tests for assessing AI models, experts have doubts about those tests’ reliability. 

Meta’s Oversight Board probes explicit AI-generated images posted on Instagram and Facebook

The Oversight Board, a semi-independent policy council for Meta, is investigating how Instagram and Facebook handled explicit, AI-generated images of public figures. In one case, an AI-generated nude image of a public figure from India was reported as pornography on Instagram, but Meta failed to remove it after two reports. The image was only taken down after the user appealed to the Oversight Board. In another case on Facebook, an explicit AI-generated image resembling a US public figure was taken down. 

Feds appoint “AI doomer” to run AI safety at US institute

The US AI Safety Institute, part of the National Institute of Standards and Technology (NIST), has appointed Paul Christiano as the head of AI safety. Christiano is a former OpenAI researcher known for his work on reinforcement learning from human feedback (RLHF) and for his prediction that there is a 50% chance AI development could end in "doom." While Christiano's research background is impressive, some fear that his appointment may encourage non-scientific thinking and compromise the institute's objectivity and integrity. Critics argue that focusing on hypothetical existential AI risks may divert attention from current AI-related issues such as ethics, bias, and privacy. Christiano's role will involve monitoring current and potential risks, conducting tests of AI models, and implementing risk mitigations. The leadership team of the safety institute also includes experts in human-AI teaming, international engagement, and human-centered AI.

• I just want to point out how funny it is that doomer has become such a commonly used term that it’s in a headline—also, I think the label is pretty clickbait-y. Use this as an example of being careful with titles if you plan on doing AI journalism :)

Why the Chinese government is sparing AI from harsh regulations—for now

The Chinese government has been sparing AI from harsh regulations, at least for now. Chinese tech giants like Alibaba and Tencent have faced scrutiny for their business practices, including antitrust violations and infringing on privacy and labor rights. However, the government has also provided support to these companies, as they are important contributors to tax revenues and employment. The government's approach to regulating the tech industry in China has been characterized by oscillations between doing too little and doing too much in various sectors, including finance and online tutoring. 

AI bots hallucinate software packages and devs download them

Generative AI models are hallucinating software packages, and developers are unknowingly downloading and installing them. Bar Lanyado, a security researcher at Lasso security, discovered that an AI model repeatedly recommended a fake package called huggingface-cli, which was then incorporated into the installation instructions for Alibaba's GraphTranslator. To test its potential as an attack vector, Lanyado uploaded a harmless proof-of-concept malware package with the same name, which received over 15,000 authentic downloads. Several large companies were found to either use or recommend the fake package in their repositories. No attacks have been identified yet, but this is another potential attack vector. 

Is robotics about to have its own ChatGPT moment?

Melissa Heikillä highlights the Stretch robot, which is a mobile robot with a camera, adjustable arm, and gripper, which can be controlled using a laptop and can perform tasks such as brushing hair and playing games. While the current capabilities of robots are limited, the article suggests that with cheap hardware, advances in AI, and data sharing, robots are becoming more competent and helpful. However, there are still challenges to overcome, such as precise control, perception of the surrounding world, and practical physics.

Papers

Daniel: The first thing I’ll mention is Penzai, a JAX research toolkit from DeepMind that lets you see model internals and inject custom logic. I think this kind of tool is awesome, and I’m planning to mess around with it more soon. 

DeepMind has a (very long!) paper out on the ethics of advanced AI assistants. 

This attempt at replicating the Chinchilla results is interesting, and it’s a little funny that the researchers have to reconstruct the paper’s data by extracting the SVG from the paper and parsing out point locations/colors—this wasn’t by choice, because the Chinchilla authors haven’t responded to their request for assistance. 

Finally, I’ll suggest this ICL paper and this preprint on the mechanics of next-token prediction. 

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this grad-student / volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Read more