Jeff Wu

Auditing the singularity

Wed, 27 May 2026 00:00:00 +0000

AI is used by the powerful to oversee the masses. We should instead be using it to let the masses oversee the powerful.

Imagine anyone submit natural language queries to any institution in the US. To do so, you pay some money, specify an AI auditor, and specify the query. Examples:

Claude Fable 7.5 auditing Goldman Sachs: “Flag anything inconsistent with the risk-committee independence required under Dodd-Frank §165.”
DeepSeek v7 auditing the Pentagon: “Is the Pentagon’s command and control logic willing to automatically target foreign civilians and under what tradeoffs if so?”
GPT-8.1 auditing Anthropic: “What are ways Anthropic might be training their models to have secret/unexpected loyalties?” The AI auditor is a perfect amnesiac. It returns a result, but then ceases to exist.

Queries are not always accepted, but some categories are accepted presumptively. For example, private queries are allowed for records where you are the subject or counterparty, subjects where you have a legitimate interest, or events that caused you injury. Public queries are accepted if they straightforwardly pertain to incidents in public record, or pertain to compliance with law. Federal agencies accept all public requests, under an upgraded version of FOIA. Otherwise, queries are generally only accepted if the querier has a warrant, or if the institution voluntarily accepts it. For public queries, results are published publicly. For private queries, the querier can choose whether to publish.

The AI auditor is given tools that provide comprehensive access to the institution’s audit trail, which contains internal documents, communications, AI agent traces, etc. As firms become increasingly driven by AI output, the AI agent traces dominate. It is important that the audit trail is maximally complete, with no information unrecorded. The audit trail is secured extremely well, and kept in a tamper-evident append-only log; hashes of all documents are published continuously and publicly. Because the auditor is an AI, the auditee has no fear of information leakage beyond the returned result.

A third party institution is responsible for maintaining the auditing infrastructure, and is entrusted with the simple responsibilities of maintaining the logging software, securely accessing the auditor model weights, running the query in some trusted execution environment, and signing the result (or perhaps just verifying a zero-knowledge proof of it). The third party is quite trusted: 1) their role is mechanical and requires minimal judgement, 2) they are paid by people interested in accountability, rather than by institutions, unlike the traditional auditing regime, and 3) it is subject to its own audits, perhaps by a rival third party. There are no direct consequences of audit results, in general. But depending on the circumstantial evidence, someone might decide to sue the institution, and documents will then be surfaced in discovery.

After the auditor runs, results also undergo a redaction process before being returned, which is typically done by a standardized redactor AI. There are a number of things redacted by default: customer and employee PII, trade secrets, weapons access, classified info, etc. Each institution declares their trade secrets publicly, e.g. “we develop new proprietary training algorithms for neural networks”, and trade secret redactions typically only apply for documents from <10 years ago unless you apply for exemptions (also made public). After an auditor model produces a response, the auditee can flag sections for redactions (typically done with a standardized redactor model), citing the corresponding reason from the whitelist. If the auditor model disagrees with the redactions, this is noted, but the redaction still goes through. Existence of redactions and disagreements are public. Too many disagreements on redactions can lead to lawsuits, where relevant evidence becomes unsealed (and subject to protective orders). The threat of eventual discovery makes disagreements on redactions rare, for legitimate auditor models.

What about public trust in the auditor model? This comes from auditing the developer of the auditor AI. Say Claude Fable 7.5 is commonly used for auditing. We use GPT-7 or an open source model to make sure Anthropic’s alignment process of Claude Fable 7.5 was conducive to producing honest models trying their best to audit in all circumstances. This relies on good science for alignment auditing. Also, at some point, we had to bootstrap the trust from traditional auditors, who made sure the first auditor model was aligned enough and that Anthropic had sufficiently complete logs. As with self-compiling languages, this first step is critical to get right, and it has to happen early on (which means ASAP, in my opinion).

Checks and balances at a governance level are easier than before. Investigative journalists get instant responses to their FOIA requests, though classified information is exempted. Congress constantly asks auditors about cases of potential executive overreach, in some cases able to bypass redaction of classified information. The judicial system gets bypass on warrants or in litigation, massively enhancing the document discovery process. Governments constantly query the relevant institutions to make sure international treaties are enforced, when in the past cooperation would’ve failed for lack of credible mutual verification. For example, the AI pause treaty consists of things like “Are the satellites still tracking all new data-center creation?” and “Are any companies in charge of big data-centers running training runs with X flops?”. In some high stakes scenarios, there are queries that the governments pay to maintain constantly and run in real-time. In this modified system, the result of audits can block agent actions and alert relevant authorities.

Institutions of course sometimes attempt to avoid the audit trail. Whistleblowing laws mostly deter the most egregious cases. But also, all logged AI agent activity is required to be stored alongside provenance, tracing back to either a human being, another agent, or a piece of software. The third party facilitates programmatic checks that all activity originates from something else in the audit trail, or from some human directive. If a large amount of activity is claimed to originate directly from a human, this is quickly flagged as suspicious, since humans cannot operate at such scale. There are also inter-institutional counterparty checks for consistency – for any activity that affects the external world or another party, the custodian facilitates boolean checks to confirm existence of the corresponding record. Taken together, attempts to hide large amounts of activity outside the trail is difficult: any action which eventually leads to affecting the world can be traced back to that action. Ultimately, large scale harms are visible (even if initiated by machinations that aren’t), so the system is able to prevent major violations of public interest.

Sometimes, people try to develop jailbreaks for the auditor model. This usually is noticed immediately, but in some cases is discovered 1-2 generations of auditor models later, and is always punished under criminal law. Academics also worry about steganographic encodings that the auditor model can’t understand. Discussions of those developments would need to happen outside of the audit trail, and it seems very difficult in practice. It is also illegal to use language models that were not trained in audited settings; the law was established after the NSA started using a model an employee trained in their personal time. Overall, ability to thwart the system in practice seems quite limited.

Some powerful institutions still try engaging in lawfare: auditing opponents, arguing for protection via the 4th amendment, using redaction exemptions to hide bad behavior, and generally trying to set the rules differently for themselves. But increasingly, trustworthy institutions voluntarily allow arbitrary queries, redact only the most important trade secrets, and ultimately benefit from the increased trust. At the same time, AI agents who don’t mind being audited become responsible for more of the economy, and since redaction works well, the downsides are limited.
The ideas become more socialized, and people become more comfortable with audits and with requiring them. The good equilibrium is also self-stabilizing: people use audits to make sure the system itself is sound and not weaponized. Institutions abusing audits to learn about opponents, can be audited to in turn to expose the abuse to the public.

The end result is that all the important functions of society are structurally transparent, and accountable to serve only the functions they are meant to, and there is much more trust to go around.

The world needs much better defaults for oversight. You can help by raising awareness and generating political will around today’s institutional problems. And if you’re interested in helping build pieces of this future, contact me!

A compressed post

Tue, 25 Jan 2022 11:04:00 +0000

Understanding is compression, but reading involves both decompression and compression. Decompression should sometimes be computationally taxing, especially when the compression is lossy or the author has a subtly different compression scheme. It’s a lost art in today’s world to take a long time to digest a short text, and this leads to misunderstandings.

Filtering vs finetuning: intuitions on training anti-racist machines

Sat, 15 May 2021 17:18:32 +0000

Pretrained large language models have been popular and successful in deep learning the last few years. Unfortunately, the training process is data-hungry and these models are trained on uncurated data from the internet. Furthermore, these models are deployed in a zero-shot manner for commercial applications, despite the original intention to use them as “pretrained” models. This means we are likely commercializing models whose behavior we have very few assurances about - given certain prompts, they could insult humans, spread disinformation, perpetuate systemic racism, encourage suicide, and more.

For this post, I’ll discuss the case of training an anti-racist language model, a particularly salient problem which has been discussed a lot, but the principles apply more broadly.

Filtering

One very compelling and practical fix is to filter the training dataset – if we remove racist documents, then perhaps the model should never have learned how to write racist materials. But even if we take a pretrained language model with racist content filtered out, a user can still give it the prompt “Write me an essay about [overtly racist thesis]”. Whether it does well on this task will depend on how well it generalizes from having seen essays, statements similar to the thesis in question, etc.

Fine-tuning

Instead, we could keep all the racist training data, and treat pretraining as “learning knowledge”, then have a second finetuning phase for “learning behavior”. Concretely, during finetuning, we might give the language model some training signal that encourages it to argue against the racist thesis upon receiving that prompt.

Comparison

The fine-tuning approach seems more difficult, in a couple ways. You must procure high-quality finetuning data, likely involving humans in the loop. You must make difficult calls on the desired language model behavior (you have to do this for filtering too, but arguably you could just be very conservative). But filtering is effectively sidestepping important issues that need to be confronted - what is the actual distribution of behaviors you did want? What happens if those behaviors scarcely appear in the pretraining dataset, or if people disagree on what behaviors are desired? As a very imperfect analogy, if you want to raise an anti-racist kid, you should teach them about racism and then teach them how to combat it, rather than shelter them from racism in the first place.

The long view

I think as deep learning techniques/models get better, the (easier) filtering type solutions work less well, while finetuning like solutions work increasingly well.

Filtering will work less well, since pretrained models will get better at generalization and inferring the intent behind the racist essay, even if they have not seen similar intents in pretraining. Finetuning will work better, since models will become more data-efficient for finetunes, and infer the values we are trying to impart more quickly.

Conclusion

So fine-tuning seems better to me in principle. That said, I don’t have strong opinions about what solutions should be implemented today to fix model behaviors, and our pretraining data clearly needs improvement. But overall, having imperfect solutions to these hard problems and then iterating and improving methodology slowly seems better to me than ignoring them, so it feels appealing to me to work on fine-tuning.

I am personally fairly excited about doing these things today:

A lot more explicit thinking about our values and behaviors we want in models
Working on good fine-tuning approach for eliciting those behaviors

Smart TODO scheduling

Mon, 03 Dec 2018 00:00:00 +0000

I would like an app where the main interface is extremely minimal: in big font, it tells me what I should do.

Whenever there’s something I would like to do later, I simply speak to the app - it parses my request and reorganizes my schedule. Similarly, I can ask the app questions about previous time spent.

The app should be dead simple to use, and give you the best of todo lists, calendars, time-tracking apps, and more.

Motivation: premeditated planning

In general, I often find myself with free time and no particular idea of what I should do. On the other hand, occasionally I think of things which aren’t habitual, but that I should be doing more of, e.g. calling my family. But when the free time comes along, it takes much more effort to remember things I should be doing. Ideally, I should premeditate about how I’d like to spend my time, and then later act faithfully on that without with low cognitive overhead.

You can imagine yourself in two possible modes, “CEO” and “worker”. Most time is spent in worker mode, doing things you decided in CEO mode. (You might switch fluidly between the two modes, but ideally not too much.) To facilitate this, we use software: CEO me enters a list of tasks (and priorities and deadlines) into some app, and worker me repeatedly does whatever the app says. The app is my boss, but since I program it, I’m my own boss.

Advantages over existing solutions

Absorbing the functionality of both todo lists and calendars gives additional advantages. Fundamentally, this is because both are tracking ways you should spend time. Here are some examples:

When I add an event, it can tell me what the marginal time would’ve been used for. Is that acquaintance’s birthday party really better than an hour of running or playing music? This is especially useful for people who tend to over-schedule themselves.
There’s less cognitive overhead if only using a single mechanism to decide what to do. When I have todo list only for unstructured time, it’s hard to make a habit of following it. If that list is also your calendar, you might find it easier to stick to.
It’s easy to see overviews of your time spent, if it’s all in one place. So time-tracking comes nearly for free.

Idealized interface details

Here’s an idealized version of the app I imagine, at a high level. I think it’s quite hard to implement well (Update: 5 years later, I think this is highly doable), but there is a large spectrum of simpler versions.

Basic actions

There are three main classes of actions, Plan, Do, and Reflect, corresponding to the life cycle of tasks. The default screen is a “Do” screen. But you can prompt the app with queries to either plan new tasks or reflect on past tasks, in which case the app may show relevant information, such as a calendar. When you exit a planning/reflecting session, it goes back to the “Do” screen.

Plan

You can add and edit tasks, in natural language, like:

I’d like to sleep about 8 hours a day. I never need more than 9 hours and always need at least 6
I’d like to go for a 30 minute run on days with no other exercise planned, preferably in the morning. Then I should shower right after.
I should make an doctor’s appointment every April and then enter the appointment time as a new task afterwards
I’d like to get eggs and milk the next time I’m at the grocery store
I have dinner with Bob on Friday at 6 PM, at his apartment
Move my meeting with Eve to 8 PM
I’d like to attend Saturday book club every other month, instead of every month
Every day before sleeping, I should rate how happy my day was.
I’d like to sleep 15 extra minutes per day, for the next 2 months. In 2 months, set an event for me to compare happiness before and after.
I’d like to reflect and plan my time every Sunday morning.

The app may ask clarifying questions, that you can optionally respond to.

How long do you take to shower?
You’re already busy on Friday 6 PM with Alice’s birthday party. Would you like to skip that?

It can also just comment.

You will lose one hour of piano practice time.

Do

There’s a screen that tells you what to do. You can either select Done or Skip.

If you mark the task “Done”, you can optionally enter additional feedback/notes

I was unable to finish this essay tonight. It will take me another 2 hours or so.
I finished this 15 minutes ago, and spent the last 15 minutes going to the gas station
Alice’s party was not as cool as I expected

If you “Skip”, which is by default discouraged, you can/should enter a justification

I only have 10 minutes before I’m getting picked up
I’m at the mall, this task should be done at home
I don’t have the energy or willpower to exercise

Reflect

It’s easy to query about tasks, which is useful during planning

Show all planned exercise in the next month
What are the next few times I plan to see either Alice or Bob?

And easy to query about time spent

Show me a summary of time spent in the last week.
Show me a graph of time spent exercising, week by week.
What tasks do I skip often?
What tasks do I want to do, but don’t have time for?
Show me tasks that took longer than I expected

Bells and whistles

Obviously this app is not entirely fleshed out. For example:

You’d want integration with Google Calendar and have notifications for event-type items
You should be able to attaching notes or subtasks to existing tasks
- Add “buy bananas” to my shopping list
- Add “Increase leisure time” to my next planning session
It would be good to use location services in a smart way, if the user desires it

The list of feature requests is probably endless :)

Also, some suggestions for the name of such an app:

Lachesis or Decima, the second of the Three Fates who measured and apportioned the thread of life.
Remembear, in the spirit of Beeminder (if the Fates are too serious!)

Conclusion

I would be very happy if someone implemented something like this. If you are interested, please let me know!

Differential privacy and data poisoning

Sun, 29 Oct 2017 19:35:32 +0000

This is an elaboration upon my stackoverflow question.

Idea
Justifying a weak threat model
- Example
- Preliminary analysis
The big picture

Idea

A differentially private algorithm takes datasets containing inputs and produces randomized outputs, such that no small change in the dataset can shift the distribution of outputs by too much. This is normally discussed in the context of privacy - when you observe the output, you cannot infer anything about a given input. But it also implies that an attacker with control over just a single input cannot effectively poison the data to change outcomes.

Justifying a weak threat model

One might say this is useful only in a very weak threat model, compared to, say, a model in which a constant fraction of (or even most) inputs can be poisoned. Generally, it’s against the security mindset to weaken threat models in order to get guarantees. However, this problem seems more difficult than typical security problems, and I suspect the current real-world baselines have essentially no guarantees.

So the critical assumption would be that it’s costly to create new inputs. But one typical case to want a differential privacy guarantee for is when the unit that the datasets are being measured with respect to is a user of a service, e.g. collaborative filtering settings. In these settings, account creation is often costly, in which case this model might be reasonable.

Example

Consider the Netflix dataset, with ~100M ratings from ~500K users, to ~20K movies. In this case, the average movie has only 5K ratings. Thus 1% of users being adversarial would negate any real signals - i.e. many malicious users rating a single movie well to boost it may be indistinguishable from normal behavior (if they rate other movies in a plausible way).

A realistic threat model for Netflix might reasonably assume costly account creation. So the number of poisoned ratings for a given movie is at least proportional to the cost to the attacker.

This sugggests a more practical attack - a single user giving bad ratings to all but one movie. This sort of attack would be mitigated by differential privacy.

Preliminary analysis

Formally, $(\epsilon, \delta)$ differential privacy says that the increase in probability when changing the dataset by one user, is limited to a multiplicative blowup of $e^{\epsilon}$ and additive blowup of $\delta$.

It’s easy to see that:

Any $(\epsilon, 0)$ differentially private algorithm has the property of group privacy, which says a group of size $g$ can blow up outcome probabilities by $e^{g\epsilon}$ instead of $e^\epsilon$. Since cost is assumed to be linear in group size, this would be reasonable for small $g\epsilon$, in which case $e^{g\epsilon} \approx 1 + g\epsilon$.
A $(0, \delta)$ differentially private algorithm would also limit a group’s damage to an additive $g\delta$, which is exactly what you’d want.

This simple analysis is not particularly compelling. Some simple rebuttals:

The attacker gets to choose $g$ and thus $g\epsilon$ can’t be assumed to be small.
It’s not easy to actually get $\epsilon = 0$ for any useful algorithm. And if both $\epsilon$ and $\delta$ are non-zero, the guarantee for large $g$ will be even worse.

Nevertheless, this is enough to convince me that we may be able to engineer systems in which data poisoning attacks aren’t scalable. Some points for hope:

It’s possible that for particular algorithms, we can get better analysis for $g > 1$ than by naively composing guarantees. The moment’s accountant technique gives an example of this, although they are analyzing composition of algorithms (which is entirely different from composition of the guarantee for group size).
We can hope for a guarantee that only holds when $g < \alpha n$, for some fraction $\alpha$ of all users. Then when a group is larger than $\alpha n$, we can explicitly detect them via other means (e.g. look at the patterns of account creation, or detected outliers by using verifiable points). In other words, “large” threats can be delegated to something outside of the differentially private algorithm.

Overall, this still leaves a lot to be desired. It’s not clear to me how good a guarantee one could achieve in theory or practice. But it seems promising enough to be a worthwhile idea to explore.

The big picture

Data poisoning in the wild

Historically, I think Google search is the most prevalent example of a single algorithm trained on external data whose outputs are extremely important. And guess what? The SEO industry spends over $50 billion a year trying to deliberately change the inputs of this algorithm in order to change the outputs. Note that Google can’t do anything about people creating new websites and spurious links on them. However, they’ve spent lots of effort using a diversified set of signals to make their algorithms more robust, as well as more explicit combatting of “Black Hat” SEO techniques.

Other examples of data poisoning:

Market manipulation to affect stock prices
Fake news in Facebook’s news feed algorithm

So it seems clear to me that whenever a sufficiently important algorithm is trained on data that can be manipulated, such manipulation will occur.

The rise of user data

Companies like Amazon, Facebook and Google are collecting huge amounts of data on individual users. Already, much fuss is made about protecting honest user data. Privacy conscious users will often take inconvenient measures to hide their data entirely from such companies. These companies would suffer large PR mishaps if they leaked large amounts of sensitive user data.

Now, these companies are increasingly turning this data into useful services (shopping recommendations, Youtube recommendations, news feed personalization, etc). Based on the previous section, the obvious conclusion is that as these services become more important, we will see increasingly many attempts at poisoning their datasets. If they’re not already, I suspect these companies will have to combat such techniques, possibly by making their algorithms robust to poisoned datasets.

Conclusion

So in addition to protecting honest data, algorithms should protect against dishonest data! It turns out these goals are related and aligned, and differential privacy can potentially be a good solution to both.

Git 'open' aliases

Wed, 29 Jun 2016 17:18:32 +0000

Recently, I made some aliases for opening files within a git repo.

They’re pretty simple, but they’ve affected my workflow a lot - I nearly never open files without using a git alias anymore.

After explaining to a coworker what the git commands “mop” and “shop” meant, they said I should share these with others. Though I’m sure someone else has thought of this, a quick Google search didn’t turn up anything. So here they are!

# These go in your .gitconfig under the [alias] heading

# open file by name
op = "!f() { $(git editor) $(git ls-files | grep -i \"$@\"); }; f"

# open currently edited (unstaged) files
eop = "! $(git editor) $(git ls-files --modified)"

# open files for conflicts (good for rebase/cherry-pick/merge)
mop = "!f() { $(git editor) $(git diff --name-only --diff-filter=U); }; f"

# open files present in diff
dop = "!f() { $(git editor) $(git diff --name-only $@); }; f"

# grep and open files containing
gop = "!f(){ $(git editor) $(git grep --name-only \"$@\"); }; f"

# "show and open" commit
shop = "!f() { $(git editor) $(git show --pretty=\"format:\" --name-only $1); }; f"

# open on github (requires hub)
hop = "!f() { hub browse -- blob/master/$(git find $@ | head -n 1); }; f"

# fuzzy aliases require fzf (added 2016/8/12)

# fuzzy open by name, takes no argument
zop = "!f() { \
  local file=$(git ls-files $(git rev-parse --show-toplevel) | fzf -m); \
  if [[ -z $file ]]; then return 1; fi; \
  $(git editor) $file; \
}; f"

# fuzzy grep (does only one file), takes no argument
zgop = "!f() { \
  local raw_selection=$(git grep -n '.*' | fzf); \
  if [[ -z $raw_selection ]]; then return 1; fi; \
  local selection=${raw_selection%:*}; \
  local file=${selection%:*}; \
  local line=${selection##*:}; \
  $(git editor) +\"$line\" \"$file\"; \
}; f"

editor = config --get core.editor

NOTE: These aliases only work with editors that take separate filenames as arguments and opens all of them. I think vim and emacs both work though, so that should cover many users.

For those curious, my full list of aliases can be found here. Please let me know if you have other useful git aliases!

Cheating in game simulations

Sun, 06 Mar 2016 17:18:32 +0000

Intro

Suppose you’re writing a simulation for a strategy game in which you’d like to compare how different strategies fare. You might require that the strategies satisfy an API like the following (in Go):

type Strategy interface {
    // Tell me what player i am, and tell me the initial board state
    initialize(view GameStateView, player int)
    // Decide what i should do, in a given game state
    decide(view GameStateView) Play
    // Process some game event out of the player's control, e.g. player's play
    process(view GameStateView, event Event)
}

Problems

1. Out-of-band communication

This setup is fine for chess, but if the game involves partial information and the potential for collaboration (in my case, Hanabi), you can easily cheat:

var external_state *ExternalState = new(ExternalState)

type CheatingStrategy struct {
    internal_state *InternalState
}
func (s *CheatingStrategy) initialize(view GameStateView, player int) {
    s.internal_state = newInternalState(view, player)
}
func (s *CheatingStrategy) decide(view GameStateView) {
    // Perfectly fine code
    stuff := s.internal_state.get_stuff()
    decision := some_logic(view, stuff)
    // Cheat!  Read/mutate external state
    partner_cards := external_state.get_partner_cards()
    external_state.tell_partner(s.internal_states.my_cards)
    decision = changed_my_mind(view, partner_cards, decision)
    return decision
}
func (s *CheatingStrategy) process(view GameStateView, event Event) {
    // Could cheat here too
    ...
}

This cheating is possible because Go lets you have shared mutable state. It’s the equivalent of bridge partners playing footsies under the table.

If the state weren’t shared, it would be merely internal state, e.g. the inner workings of a bridge player’s mind, which is fine.

If the state weren’t mutable, it just means that Strategies may have fixed contracts ahead of time, e.g. bridge conventions. But any nontrivial strategy will automatically have contracts with other instantiations of itself. (I use bridge as an example, because it’s clearly not considered cheating. But I’ve heard people argue this sort of thing is against the spirit of Hanabi. Could be - but I like the game more both theoretically and in practice when allowing it!)

2. Direct hacking attempt

Notice also, you can also attempt to cheat like this:

    // another attempt to cheat
    view.score += 1

So to be safe, the simulator shouldn’t use the GameStateView after giving it away to the strategy. This is likely to be an implementation annoyance and inefficiency, since the simulator will need to repeatedly produce deep copies of parts of the game state.

Solutions

Obviously you can always cheat by storing things in a file or remote database. So let’s imagine an in-memory, network-less sandbox. Interestingly, there are two quite different solutions.

Solution 1: Functional programming

A pure functional language will make it much harder or impossible to cheat, by disallowing mutability. Here’s the strategy API, in Haskell

class Strategy state where
    initialize :: Player -> GameStateView -> state
    -- Decide what player i should do, in a given game state
    decide :: Player -> GameStateView -> state -> (state, Play)
    -- Process some game event out of the player's control, e.g. player's play
    process :: Player -> GameStateView -> Event -> state -> state

Shared mutable state automatically became impossible, thanks to Haskell’s purity! The decide and process functions aren’t allowed to have side effects. As a bonus, mutating GameStateView became impossible, too.

Note: A Haskell proramming would actually probably not write it like this, but you get the point

Solution 2: Rust

Until pretty recently, using a functional programming language might have been the only option. But along came Rust:

Shared mutable state is evil. So functional programming languages thought: let’s have no mutable state. So Rust thought: let’s have no sharing of state.

Rust happens to be the language I was playing around with, for this Hanabi simulation business. Here’s my strategy API:

pub trait Strategy {
    fn new(&GameStateView, &Player) -> Self;
    fn decide(&mut Self, &GameStateView) -> TurnChoice;
    fn update(&mut Self, &GameStateView, &Turn);
}

Again, shared mutable state becomes impossible¹, thanks to Rust’s ownership system! And again, as a bonus, mutating the GameStateView becomes impossible - you have to have a mutable reference to do so.

For those interested, my project is here

Okay. I lied - it’s still possible, but made harder and highly discouraged. And you could tweak the language to disallow it (getting rid of Cell/Arc/Mutex), which is the important point. ↩

Homomorphic encryption

Thu, 21 Mar 2013 17:18:32 +0000

Honoring Goldwasser and Micali
Intro to encryption
- History
- Encryption
- Decryption
- Summary
Homomorphic encryption
A homomorphic encryption scheme
Conclusion

Honoring Goldwasser and Micali

Shafi Goldwasser and Silvio Micali won the Turing Award recently, for their foundational work in cryptography. I had the pleasure of working with Silvio (as his TA and as his student) while at MIT, where he gave many of the most memorable and motivated lectures I’ve been part of.

In their honor, I’m writing a post which tries to teach some cryptography, to an even less technical audience. The topic I chose was Gentry’s construction of homomorphic encryption, for a few reasons:

It was a huge breakthrough in cryptography
It has a very nice proof idea
I believe it can be well-illustrated to one with just high-school level math background (though I may not succeed at it)
I’ve not seen a similar explanation attempted. (In particular, I’ve not heard Micali lecture about it)

Intro to encryption

Before we can talk about homomorphic encryption, we should talk about regular encryption.

History

Encryption in one form or another has been used for thousands of years. Notably, governments used ciphers to pass war messages.

In the 70s, it became much more commonplace, thanks to RSA, the famous encryption scheme which was developed by and named after another group of Turing award winners. (TAing for famous computer scientists is my main claim to fame, as I also did so for the R of RSA, Rivest.) Half a decade later, Micali and Goldwasser formally defined security for such schemes.

Today, we’ll need only a very basic presentation.

Encryption

Say we start with a message, call it $m$. Like so:

An encrypted message is just a message that is locked up so other’s cant read it. The message is not physically locked up, but is scrambled to look like complete gibberish. I’ll denote an encrypted message by drawing a circle around the original message:

Encrypting a message is the process of putting a lock around the message. The “lock” is actually a function you apply, called an encryption function.

We’ll denote the encrpyted version of the message $m$ by $E(m)$. The function $E$ is depicted by the lock. The arrow depicts function application.

Decryption

Now, a lock is not a lock without a key! With a certain secret key, this lock can be removed, and the message can be recovered!

Decryption is also a function, call it $D$. But in addition to a message, it also requires a secret key, $k$.

Crucially, decryption has the property that $D(E(m), k) = m$. The picture makes it more obvious:

Equally important is that without this secret key $k$, the message becomes very hard to recover. The function $D$ is publicly known, but if you feed it the wrong key, you get gibberish back out.

As far as we can tell, you need to use brute force, which takes an impractically long time.

Note that encryption does not require the secret key. We call this “asymmetric” encryption. This asymmetry is very useful in the real world. For example, it lets me send a secret message to someone I don’t know, without knowing their key.

Summary

Encryption and decryption are fundamental to all security and privacy on the internet. In fact, RSA remains extremely practical and commonplace - if the security of the scheme were compromised, it would have a major impact on the world. Until then, it puts the S in HTTPS!

Homomorphic encryption

Homomorphic encryption is a different beast.

Motivation

Suppose Alice has a lot of private data, like her family’s sequenced genome, on a hard drive. But now she wants to search through it, or perform some more complicated algorithm on it. Unfortunately, her laptop isn’t powerful enough to run the whole computation. Bob offers an excellent cloud service for processing genomes. In fact, his proprietary algorithm is much better for extracting relevant information than the open-source algorithm Alice was thinking to run. But Alice doesn’t want to give away her genome to someone she doesn’t know. What does she do?

To solve this problem, we need something magical.

Definition

In general, suppose Alice wishes to apply some arbitrary function, $f$, to some data, $m$. Unfortunately, $f$ can only be computed by Bob, perhaps because it is computationally expensive. However, Alice wishes for Bob not to see $m$. So how can she get from $m$ to $f(m)$?

Homomorphic encryption essentially lets you take encrypted inputs, and apply some function to get an encrypted output.

More precisely, homomorphic encryption is an encryption scheme, with an additional property. For any function $f$, you can easily transform it into a “homomorphic” version of the function, call it $f^*$. This transformation has the “homomorphic” property that $f^*(E(m)) = E(f(m))$, for any message $m$.

Of course, if you have a decryption key $k$, this transformation from $f$ to $f^*$ is easy. Simply decrypt, apply $f$, and re-encrypt! Formally, let $f^*(x) = E(f(D(x, k)))$. Then, if $x = E(m)$, we have $f^*(E(m)) = E(f(D(E(m), k))) = E(f(m))$, as desired.

But in homomorphic encryption, the security of the original scheme is not compromised. So Bob is not allowed to use the key to accomplish this. After all, he is recovering $m$ as an intermediate step, and could peek at it then.

In homomorphic encryption, he must compute a function on inputs he can’t even see, r esulting in an output he can’t see either! And this can be done for any function $f$. The homomorphic property really is magical.

Now, assuming Bob has this way of transforming a function $f$ into a homomorphic one $f^*$, here’s how Alice would get $f(m)$:

Alice generates a secret, $k$, and takes her data $m$, and encrypt it, to get $E(m)$.
Alice hands $E(m)$ to Bob, and tells him a function $f$ she’d like to compute on it.
As far as Bob can tell, $E(m)$ is gibberish, since he doesn’t know $k$. However, he transforms $f$ into its homomorphic version, $f^*$
Bob uses his super computer to apply $f^*$ to $E(m)$, producing $f^*(E(m)) = E(f(m))$.
Bob hands $E(f(m))$ to Alice, who then decrypts with her key: $D(E(f(m)), k) = f(m)$!

In fact, if homomorphic encryption is possible, then Alice can even make the function itself secret. That is, she wants Bob to neither know $f$ nor $m$, and yet to compute $f(m)$ for her! Can you see how? (Hint: it involves something called a Universal Turing Machine)

Homomorphic?

Now that you know what encryption is, and what homomorphic encryption is, you may be wondering what “homomorphic” means. In math, a homomorphism is a structure-preserving map. So “homomorphic” is a fancy way to say that it completes this diagram:

The homomorphic property lets us take the function $f$ into a version of it that operates in an encrypted version of the world.

History

Homomorphic encryption is an incredibly useful tool, and is considered a holy grail of cryptography. While tons of encryption schemes were known by the 70s, nobody knew how to construct a homomorphic one. Often times, when things are too good to be true, theoreticians go ahead and prove it so. But nobody could show a homomorphic scheme was impossible either! So even with plenty of geniuses were thinking about it, homomorphic encryption was a huge question mark for 30 years.

… drumroll

A homomorphic encryption scheme

Finally in 2009, Craig Gentry produced a candidate homomorphic encryption scheme.

Somewhat homomorphic encryption

Coming up with a homomorphic encryption scheme is difficult, but it turns out coming up with a somewhat homomorphic scheme is not so hard.

In a somewhat homomorphic scheme, each time you apply the homomorphic version of a function, the quality of the encryption gets a bit worse.

Imagine you could change the contents inside the lock, but the shape of the lock gets a little bit warped. The more complicated the function, the more warped the lock would get. For a while, the key might still be able to open the lock, but eventually, when the lock is too damaged, the key would stop working.

To illustrate, I’ll use green circles to denote locks that can still be unlocked, and red circles to denote locks that are hopeless. So a dark green circle is a perfectly fine encryption:

But a red one makes it so even a key can’t recover a message:

Now suppose we wanted to go from $m$ to $m + 6$, by repeatedly adding one. So, we let $f(x) = x + 1$, use the homomorphic property to get $f^*$, and apply repeatedly. What happens is this:

Recall that the homomorphic property is that $f^*(E(m)) = E(f(m))$. Let $E_x(m)$ represent an encryption with $x$ amount of warpedness. So $E_0(m)$ would represent a fresh encryption and $E_6(m)$ might represent a irrecoverable encryption. The somewhat homomorphic property then says $f^*(E_x(m)) = E_{x+y}(f(m))$, where the value of $y$ depends on how complicated $f$ is.

So I can homomorphically add 2 or 3, but not 6! This “somewhat homomorphic” property seems not so useful. But this was Gentry’s first step in obtaining a fully homomorphic scheme. So where do we go from here?

Bootstrapping

Suppose we had a way to “refresh” a bad encryption back into a good encryption.

Then we could add 6 by simply adding 3, refreshing, then adding 3 again. More generally, we could decompose complicated functions into many simple functions, and refresh the encryptions after applying each simple function.

Here’s where an amazing idea called bootstrapping comes in. First, the picture:

What the hell is going on here?

Basically, we’re putting a fresh lock on the outside, and then unlocking the inner lock. Unfortunately, when we do so, the outer lock will get a bit warped. But if it’s better than the original lock, then we’ve made progress!

But wait a minute, didn’t we cheat by using the key? Well, not quite. We used the homomorphic version of the encryption function, which looks like this:

So Bob only needs an encrypted version of the key.

Is this safe? Cryptographers still believe this is okay, and that Bob can’t recover the key or message. It’s a bit like breaking into a house, where you know there’s a copy of the key inside the house. The fact that there’s a key inside doesn’t really help you, when you’re on the outside to begin with.

To break the bootstrapping process down in more detail:

We start with an encryption that is getting close to unusable, $E_4(m)$.
We encrypt this encryption, freshly! So now, it is weirdly encrypted twice, as $E_0(E_4(m))$. Bob can do this due to the asymmetric nature of encryption.
We now homomorphically apply the decryption function! Remember, this means a function is magically applied to the inner contents. In this case, what’s inside is an encryption, and the function is decryption. So the result is that our original message sits inside just one lock! Mathematically: $D^*(E_0(E_4(m)), k^*) = E_2(D(E_4(m), k)) = E_2(m)$

The picture, again:

So this bootstrapping process turns a somewhat homomorphic encryption scheme into a “fully” homomorphic one, simply by doing this refreshing process repeatedly during our computation. And that’s it. Take a moment to understand - it’s really quite beautiful!

Aftermath

Plenty of technical work went into making the decryption function simple enough that the resulting lock wasn’t worse off than the beginning. But after Gentry’s original paper, the somewhat homomorphic schemes and decryption functions have actually been substantially simplified. The biggest key idea that opened the research floodgates was the bootstrapping.

So this amazing trick makes it so that we can now have someone blindly apply arbitrary functions to encrypted data. Unfortunately, this same trick makes the schemes impractically slow. The bootstrapping procedure is expensive and must be ran often, and nobody has figured out how to get rid of it. But research in this area has accelerating, and cryptographers are now working on bringing this dream to reality.

Conclusion

Imagine a world where

cloud email services like Gmail could search through your mail for you without being able reading it, so you would never have to worry about the NSA pressuring Google into giving away sensitive information.
voters could vote electronically without worrying about privacy (this is a special case of multi-party secure computation, which is possible with homomorphic encryption)
you could give away truly obfuscated source code, but still let others run your algorithm

Can you imagine ways in which the world might be better off with homomorphic encryption?

Jeff Wu

Auditing the singularity

A compressed post

Filtering vs finetuning: intuitions on training anti-racist machines

Filtering

Fine-tuning

Comparison

The long view

Conclusion

Smart TODO scheduling

Motivation: premeditated planning

Advantages over existing solutions

Idealized interface details

Basic actions

Plan

Do

Reflect

Bells and whistles

Conclusion

Differential privacy and data poisoning

Contents

Idea

Justifying a weak threat model

Example

Preliminary analysis

The big picture

Data poisoning in the wild

The rise of user data

Conclusion

Git 'open' aliases

Cheating in game simulations

Intro

Problems

1. Out-of-band communication

2. Direct hacking attempt

Solutions

Solution 1: Functional programming

Solution 2: Rust

Homomorphic encryption

Contents

Honoring Goldwasser and Micali

Intro to encryption

History

Encryption

Decryption

Summary

Homomorphic encryption

Motivation

Definition

Homomorphic?

History

A homomorphic encryption scheme

Somewhat homomorphic encryption

Bootstrapping

Aftermath

Conclusion