Emerging Challenges Fund (2023)

for research directions focused on safely building advanced AI systems

What FAR produces

FAR AI conducts and supports research in several areas. These currently include:

• Robustness: Modern AI systems are easily exploited by other AI systems specifically trained to do so. For example, FAR and their collaborators were able to cheaply create an AI system to reliably beat a state-of-the-art Go-playing AI. FAR seeks to measure the severity of this problem, and how it will change with model size and with different countermeasures.
• Value alignment: How can AI systems be trained to do nuanced tasks that humans care about, rather than maximise simple metrics? For example, FAR collaborators developed a method for training a language model by giving it feedback in natural language. This radically increases the efficiency with which humans can train a model to behave as intended.
• Model testing: Finding the ways that advanced AI systems fail. For example, FAR collaborators ran a competition to find cases where larger models do worse than smaller models. This and following work found very few examples.
• Interpretability: Making it possible to understand what is going on inside AI systems. For example, FAR and collaborators developed a prototype for constraining the inner workings of a neural network to a relatively small number of possibilities, with limited impact on accuracy.

For further reading: FAR recently published updates on their research in 2023.

FAR also conducts other projects which are unlikely to be supported by this grant. These include:

• Work supporting the overall field, such as co-hosting the recent International Dialogue on AI Safety, which brought together top scientists from China and the West.
• An AI coworking space with about 30 members.

For further reading: FAR recently published updates on their overall operations.

How FAR works

FAR employs a technical team of research engineers and research scientists. A majority of what FAR would do with an increased budget is to hire 2 more research engineers, enabling them to take on more and larger projects. These are likely to be primarily in-house projects. FAR also collaborates with external researchers, especially those working in academia, who usually cannot employ research engineers.

FAR also employs communications staff. They improve outputs by, for example, making papers clearer and publishing summaries. This makes it easier for other researchers to build on the work of FAR and the external researchers FAR supports. Communications staff also produce some unique communications projects such as clearly explaining the pace of AI progress.

Key considerations

The key reason to support FAR is that the organisation has produced a range of research outputs which lay the groundwork for building safer AI systems. We reviewed a sample of past outputs and saw that they were typically targeted at important unsolved problems. Additionally, FAR’s projects also seem to have been executed very well, with papers having been accepted into top conferences, won minor prizes and/or been cited many times.

Our main concern is simply that additional funding might be used for less promising projects. In particular, moderate staff turnover means that future projects might be different. For example, FAR affiliates have departed to continue their work at Anthropic or EleutherAI. Even so, past projects are evidence of FAR’s ability to select strong projects to support and incubate.

to commission research into the efficacy for reducing disease transmission, eye safety, and impacts on air quality of far-UVC light

Reducing the spread of infectious disease

Certain wavelengths of germicidal light called ‘far-UVC’ inactivate pathogens in the air and on surfaces while appearing to be safe for humans. Could this be a powerful tool for reducing the spread of infectious disease, and even stop or slow the next outbreak which could become a pandemic? This funding will be used to commission research into the safety profile of far-UVC, making it possible to understand how best to deploy it.

Blueprint Biosecurity is making calls for proposals that most efficiently and comprehensively address the outstanding questions. This funding is likely to be used to answer questions in the areas of eye safety and air quality.

Testing eye safety

When light shines on people, most of it is absorbed or reflected by their clothes. However, it is important that this kind of light is safe for people’s skin and eyes.

• There is already very good evidence that most far-UVC light is absorbed by the outer layers of the skin. These cells have no nucleus (where the DNA is), so, unlike other wavelengths of UV light, there is no risk of DNA damage and consequent cancer risk. For example, see Görlitz M, Justen L, Rochette PJ, et al. 2023, page 6.
• However, the most up-to-date standards for occupational use in the USA have a limit for eyes that is around 3 times lower than for skin, and there is relatively much less evidence for eye limits and safety in general. There is anecdotal evidence that far-UVC above certain levels can cause discomfort in the eye, even if there are no signs of long-lasting harm.

This funding will be used to study the question with the most important remaining uncertainties. That is photokeratitis, a temporary but painful condition of the outermost layer of the eye (the cornea). Improper installation of other UV technologies (not far-UVC) has caused photokeratitis in the past. Blueprint Biosecurity has now published its first request for proposals, seeking experiments to:

• “Determine the levels, if any, at which irritation/discomfort is experienced under far-UVC exposure, and how quickly any such symptoms disappear after exposure”;

• “Establish mechanism and cause of possible human eye irritation/discomfort under far-UVC exposure”; and

• “Develop satisfactory and sufficient models to predict possible far-UVC-induced effects that contribute to irritation/discomfort mechanisms in human eyes”.

Having a clear picture of the risk will help with:

• Knowing which ways of using far-UVC make sense. (This would, in turn, inform which far-UVC grants make sense, and how much to fund far-UVC work in general.)
• Knowing what regulatory limits make sense. (To get rid of as many germs as possible, the limits should be set as high as is safe, but no higher.)

Testing impacts on air quality

Beyond what happens when far-UVC light shines directly on people, Blueprint Biosecurity is investigating the reactions far-UVC causes (in small quantities) in the air, and whether they cause a problematic buildup of anything harmful.

Specifically:

• Far-UVC causes the production of a small amount of ozone (O₃) and hydroxyl radicals (OH) from the oxygen and water in the air.
• These molecules react with volatile organic compounds in the air (e.g. limonene, which gives the lemon smell to many cleaning and other fragrant products) to form secondary organic aerosols. These are a type of fine particulate matter, which is a risk factor for health issues, mostly lung-related.
• far-UVC photons also interact directly with some volatile organic compounds, but this is not well studied.

The most valuable experiments to commission for this purpose could involve:

• Measuring how much ozone and fine particulate matter is produced in situations representative of real-life far-UVC deployments;
• Measuring the formation of secondary organic aerosols for the most concerning/prevalent volatile organic compounds in controlled experiments;
• Replicating past studies with surprising results to better understand them — such as one experiment where far-UVC caused much more pathogen inactivation and ozone production than expected;
• Better quantifying the potential health impacts of increased indoor ozone (all estimates currently trace back to one 2016 paper on outdoor ozone);
• Identifying mitigation strategies (e.g. removal as part of the ventilation system, ozone scrubbers, or benign materials that are known to be effective ozone sinks, such as clay); or
• Identifying new sensor technologies, as it is currently prohibitively expensive to monitor ozone with the level of accuracy that may be necessary.

If you might apply when Blueprint Biosecurity publishes a call for proposals on air quality, please do not take the above list as definitive. Blueprint Biosecurity is in the process of refining it.

As with eye safety, studying these issues will help understand what uses of far-UVC make sense.

Key considerations

A key motivation for this grant in particular is that Blueprint Biosecurity may be the best-placed group in the world to ensure that a wide range of experiments address the most important questions about far-UVC.

• In our experience, Blueprint Biosecurity is very focused on ensuring that its work is action-relevant.
• Blueprint Biosecurity has been working to identify the research outputs which would result in the most compelling and useful evidence on the photokeratitis question. We have read some of their internal notes and found them to be the most informative resources we have found on some of these issues.
• Blueprint Biosecurity will work with external researchers to help ensure that their experiments will answer our key questions.

Our key remaining uncertainty is simply how promising far-UVC will ultimately be in slowing down the worst future pandemics.

• It might be the case that far-UVC is impractical to deploy in sufficient intensity to have sufficient effect on transmission, or that this takeup does not happen for reasons unrelated to the safety and societal value.
• In particular, far-UVC is currently too expensive to deploy in a significant fraction of indoor spaces. This means that hardware R&D to produce far-UVC affordably is important. For the most part, this will likely be best done by companies.
• Experiments like those funded by this grant will help clarify other uncertainties like what intensities of light are very safe.

for research into nuclear decision-making

Understanding and improving decision-making around nuclear crises

Decision protocols around nuclear weapons were established in the middle of the 20th century, when we had just a basic understanding of how people make decisions under time pressure and uncertainty. This project will share existing research and generate new research to help government officials review their plans and procedures in light of the latest science on decision-making under pressure. Military and civilian leaders can apply these insights to improve decision processes and training exercises.

Decision Research, in partnership with Brown University, will compile and present insights from decision-making research relevant to nuclear protocols and doctrine. The team includes four scholars (Professors Paul Slovic, Rose McDermott, Reid Pauly, and Commander Daniel Post) who have made impressive academic and public policy contributions. The project will commission new survey research to fill gaps and pinpoint factors that could compromise decision-making. The team is in conversation with U.S. civilian and military leadership, providing an avenue to inform policy.

Key activities

Activities funded by this grant include:

• Meetings with policymakers to present information and understand the problems they face. These meetings can introduce policymakers to underappreciated dynamics on psychic numbing, cognitive shortcuts, and the influence of decision framing. They will help researchers design studies that capture real-world dynamics.
• Conducting new survey research to gain insights into how U.S. policy and military elites reason about what motivates and deters potential adversaries.

Path to impact

This research and analysis can help improve:

• Decision-making processes, the human element of nuclear command and control systems.
• Guidance on nuclear posture and force structure to improve resilience during crises.
• Training exercises and simulations for the president, top-level civilian leadership, and military strike advisors in Strategic Command and the Joint Staff.

The ultimate goal is to incorporate the latest insights from cognitive science into nuclear decision processes in order to reduce the risk of catastrophic failure.

In short: this fund is now fully managed by Longview Philanthropy, but will continue to support the same kinds of work. On Tuesday 28 November 2023, Giving What We Can emailed all past donors to the fund to share this information.

Our collaborators at Giving What We Can initiated this fund as a clear way for anyone to contribute to work protecting future generations, as described here as the Longtermism Fund. Initially, Giving What We Can asked Longview to recommend grants for the fund to make. Now, Giving What We Can is investigating which evaluators to rely on (the “evaluating evaluators” project). Giving What We Can wanted to do this as impartially as possible. To reduce conflicts of interest, they therefore asked Longview to take sole responsibility for managing the fund going forward.

Since Longview was already recommending grants to the fund, the main change is simply that Longview will publish grant reports, rather than Giving What We Can publishing reports with help from Longview. Reports will be published on our website. (The finances of the fund will continue to be managed through Giving What We Can’s donation platform.)

We are taking this opportunity to clarify the fund’s aims:

• The original announcement stated that the fund will support work that “Reduces existential and catastrophic risks, such as those coming from misaligned artificial intelligence, pandemics, and nuclear war.” The fund has done this and will continue to do so.
• The announcement also said that the fund would support work that “Promotes, improves, and implements key longtermist ideas”. In practice, we have not recommended any grants which fall solely into this category. Therefore, we will remove this item from the fund’s aims.
• We are renaming the fund to the Emerging Challenges Fund to put the focus on the issues that the fund tackles, which we have found is overall clearer.

These updates clarify how the fund already works. They do not represent a material change to the fund’s activities.

We anticipate publishing the next report on this fund’s activities in mid-2024.