λk.(k blog): λk.(k blog)

Please don't generate your bibliography

2026-03-25T00:26:51Z

I read a lot of bibliographies. I also read a lot of bad bibliographies, most of which are clearly autogenerated. I don’t like reading bad bibliographies.

There are two purposes to a bibliography.

The first is to communicate which article is being cited to the reader. For that, it’s important to make the citation information very clear and consistent.

The second is to make it possible for the reader to find the thing being cited. For that, you need as much information as necessary to track down the original article. If you have a DOI, that becomes pretty trivial, so most other information can be excluded. If you don’t have a DOI, you may need to call a librarian and they may need lots of specific information. If you haven’t had to do this, go find an old paper in your field and try to track down a physical copy. It’s a learning experience.

I have a style guide and some rules-of-thumb I use to try to satisfy these two goals:

For provenance:

For all citations, include DOIs, or URLs from the most authoritative, least-likely to fail/change institution if no DOIs available.
If no DOI/URL is available, include as much citation information as possible.
When DOIs and canonical URLs are available, avoid including things like publisher names and addresses, editor names, etc. This is just clutter.
If a URL of dubious longevity is the only available, try to archive it with https://web.archive.org/.

For clarity:

Never use automatically generated bib information; always clean it up and try to keep consistent.
- I always manually replace the conference name with something more legible. The autogenerated one will often also contain the venue location, the date, the venue’s iteration number. I exclude all that, since most of that is elsewhere in the bib or is irrelevant. You can use bibtex string constants to make this easy.
- I exclude the ACM SIGPLAN nonsense for the most well-known venues.
- I typically exclude editors, which don’t really matter
- I typically exclude page number, except for journal articles
- I typically exclude the publisher, unless there is no DOI or URL. The publisher is useful if you need to track down a copy of the article, but if you have a DOI or authoritative URL, the publisher shouldn’t be necessary.
Avoid duplicate DOIs + URL.
Use braces {} to guard any word in titles that need capitalization, such as {AI} or {CIC}.
Never use the SIGPLAN Notices version of a citation. Some versions of ICFP, POPL, etc, papers were also published as “SIGPLAN Notices” journal articles. This can confuse readers. The two versions are the same paper, but have different DOIs and citation information. Make sure to find the conference version, and not the SIGPLAN Notices version. For example, see these two citations:
- https://dl.acm.org/doi/10.1145/3093333.3009886
- https://dl.acm.org/doi/10.1145/3009837.3009886

To implement these rules, I use Jabref. It can pull in the autogenerated information, then I manually clean it up according to my rules. Jabref supports marking fields as optional or required depending on the type, so I usually just delete all the optional fields, unless there’s not DOI. I use lots of bibtex string constants for common venues, to help clean up the bibtex and keep things consistent.

It’s always disappointing to me when I read a completely autogenerated bibliography. It’s obvious; it looks like slop, and accomplishes neither of the two goals of a bibliography. I was just reading one, and no entry had a DOI at all, titles had incorrect capitalization all over the place, many of the proceedings included cities and dates duplicated in the date field.

Against Vibes Part 2: Ought You Use a Generative Model

2026-03-13T17:13:06Z

Since the wide-spread availability and forced deployment of generative models, people have argued about the ethics of using them. Many arguments have been presented to argue that they’re bad: they use too much electricity, boil the oceans, massively infringe on copyright, put people out of work, and generate slop. And therefore, you are ethically obliged to not use them.

I sympathize with the fundamental conclusion: that there is something ethically bad about the current situation. But I can’t really take these arguments seriously, for two reasons: (1) if you look into any one of the arguments, the details are (shocking) a little more complicated (2) it’s hard to judge the validity of an ethical argument when no one is willing to make explicit their system of ethics.

In this post, I’m not going to argue about whether generative models are useful. I came up with a model of usefulness in the previous post, which doesn’t say generative models are useful, but gives you a framework for deciding whether one is for a given task and user. For the moment, though, I’ll assume for the sake of argument that there is a use for generative models, so now we need to answer a different question.

Ought you use a generative model? I don’t mean is it good for a specific task; that’s an is question, about whether it is useful. I mean ought you use a model: is there an ethical argument for or against you using generative models, categorically. I’m not going to spoil the ending, because the argument is more important than my conclusion.

An Ethical Framework

It is slightly infuriating to me that people will argue about ethics without interrogating their own ethical axioms, or those of their interlocutor. It makes so many ethical arguments seem like a series of randomly generated sentences.

The dominant ethics pretty much everywhere is Utilitarianism. Utilitarianism makes ethical decisions by not supposing apriori what is good and bad, but instead that everyone seeks some generalized notion of utility. Maybe you get utility from eating delicious baked goods, but you lose utility from having to work a soul-crushing 9—5 job. An ethical action, under this framework, is the one that maximizes utility, or in negative utilitarianism, minimizes negative utility.

This is a stupid ethical framework. It falls prey to various paradoxes, like stupid trolley problems and the utility monster who derives more utility from harming you than you lose by being harmed. It presupposes that “utility” is quantifiable, and ignores standard problems that arise in optimization problems, like that to optimize one variable you must both ignore and therefore minimize other variables. As a consequentialist framework, it falls prey to many of the standard problems of trying to predict the consequences of one’s actions, but as a framework that pretends to be very mathematical, it carries these problems out to infinity. This leads to bonkers philosophies like Effective Altruism and Longtermism, which support absurd beliefs like that to maximize utility we must rush headlong to develop a general purpose artificial intelligence that is the utility monster and maximize its utility.

But if I’m honest, I think my fundamental problem with it is that it exists to justify harm—to say, “yes, actually doing harm is the ethically correct choice”. That feels wrong.

So let me start with my ethical axioms, my framework for making ethical decisions. There are many like it, but this one is mine.

I’m some kind of hedonist: I believe that which is enjoyable is ethically good, and that which is harmful is ethically bad. I believe one should take actions that do not cause harm, and ideally take actions that are also enjoyable.

I believe one is ethically obligated not to cause harm, but has no positive obligation to cause pleasure; it is actually a kind of negative hedonism. But if you can cause pleasure, that is good.

My hedonism is stratified: base (Type 0) good things (e.g., deriving pleasure from eating a pastry) are good; base bad things (e.g., intentionally causing harm to an innocent being) are bad. Deriving pleasure from a (Type 0) harm, sadism, is a (Type 1) pleasure that is bad. This is ethically distinct from deriving a (Type 0) pleasure merely at the expense of something else, like eating a cruelly harmed animal, which is also bad but for a different reason. I have notes on further levels somewhere, but I think these suffice for the purposes of my arguments.

I reject that “good” and “bad” are quantifiable. I cannot decide, ethically, whether one harm outweighs another. Harm always outweighs pleasure; there is no “quantity”. “Is it more ethical to kill one man to save two men?” what a stupid question.

Instead, my framework is incomplete: some actions are neither ethical nor unethical. For example, if you are required to act and any choice you make causes harm, this framework does not help you, except in one way: it decides no action you take is unethical, since there is no ethical choice. You are only ethically obligated to act in a particular way when you have a choice between an action that causes harm, and an action that does not cause harm. “Should you cause harm to save a life?”, well, if failing to take action results in harm, and your action results in harm, then either action is equivalent under my framework. I’m happier to admit ignorance than to admit paradoxes.

I also subscribe to some kind of bounded rationality. It is entirely pointless to try to consider all consequences of your actions out to infinity, taking into account all possible information. I am only obligated to act on information I can reasonably be expected to have, and with respect to consequences that are reasonably foreseeable. It also entirely pointless to try to consider all possible actions one could take; you aren’t actually infinitely capable. As things get more and more complex, and less and less certain, I say: who knows, try your best.

So there’s my framework: a stratified anti-quantitative bounded negative hedonism. Now that we have a framework, lets start looking at ethical arguments about generative models.

The Resource Argument

One argument against generative models is that these models “use too many resources”, typically electricity or water. So what are the ethics of this argument?

Using electricity, in itself, does not cause harm. There are many examples of using electricity bringing joy and happiness to people; those uses are good. There are certainly uses to which electricity can be put that are unethical: I could electrocute you, causing you harm (unless you’re into that). But that’s not the electricity causing harm; that would be me causing harm.

The argument about electricity is probably about the effect on climate change, since increased electricity use probably means increased carbon emissions, which cause harm. So using a generative model, and therefore increasing electricity use, and therefore maybe increasing emissions, may cause harm.

This is not a strong argument. The source of electricity matters, for one. If all or most electricity came from renewables, there would be no harm. Increasingly, renewables are becoming a dominant mode of electricity generation.

Given the uncertainty involved, I don’t think this on its own creates an ethical obligation on individuals to use or not use generative models.

Worse, the argument is contingent on the technical capabilities of generative models. If they become more efficient, do they become ethical?

There are other other forms of this argument, but they aren’t really about resources. For example, one might argue: they use more electricity than alternatives and produce worse results. I’ll call this a “slop argument”; it’s not really about resource use, but about capabilities. This implies it’s okay to use more electricity, and therefore possibly worsen climate change, as long as the generative model is good enough. Any such argument is doomed to failure on many fronts, such as the booster’s favourite argument that the models are going to get better and better until they eventually achieve AGI. However, its fundamental flaw in my mind is that it’s utilitarian: it justifies harm.

There’s one other form of this argument, which shifts the conversation from the mere resource cost to the cost of data centres or the industry as a whole. This is also not really a resource argument: I’ll call it “the power argument”, and address it separately. Again, a data centre using electricity doesn’t necessarily cause harm, and even if it did, it doesn’t tell us whether an individual ought to use a generative model, since their action has no effect on the deployment of new data centres and therefore the increased resource usage.

So in short, I don’t think there is an ethical imperative to not use generative models because using them increases resource consumption.

The Intellectual Property Argument

One argument I find super weird is the intellectual property argument: training generative models has “stolen” tons of work, or reproduces copyrighted work. This is particularly weird when it comes from people who normally don’t give a shit about intellectual property, consuming much of their media via pirate sites or torrents.

Worse, it’s not even clear whether this is infringement of intellectual property rights. Downloading copyrighted material isn’t a violation; only reproduction is. It’s very clear that some models will reproduce stuff from their training data, so you might argue there is infringement there, but it’s not clear. Sometimes reproduction is “fair use”. Reproduction for the purposes of commentary or criticism is usually fair use. The argument for and against has to do with demonstrating economic harm to the rights holder, which might be difficult. Do you think economic harm has been caused to book publishers as a result of training generative models? I kind of doubt it, but maybe. And it’s not what generative models are supposed to be doing, which makes the argument harder. You can genuinely argue that reproduction is a bug to be fixed.

Even if it were infringement, while infringement might be illegal, that doesn’t make it unethical. Who is harmed if I pirate all of The Fast and Furious movies, or if I train a model that reproduces GPL source code?

Ignoring whether or not it even is infringement, intellectual property itself isn’t necessarily good. Why is it ethically good to give a person a complete monopoly on the production of something? And for copyright, that monopoly is crazy. 70 years after the death of the original author? What kind of creativity does that inspire? Why should we give such power to one individual, to legally go after anyone who wants to compete at the production of a good idea whose author is long dead? There’s nothing ethical about this, and in fact, I firmly believe intellectual property as it is currently implemented causes significant harm.

So, no, I don’t think you have an ethical imperative to avoid generative models because they might infringe copyright.

The Slop Argument

A very common complaint, and sort of argument, is that generative models produce slop. That is, they produce output that is not of high quality; it doesn’t meet the requirements or goals of the output. In technical work, it doesn’t meet engineering standards or design requirements. In creative work, it fails to express anything of artist interest or intent.

This is not really an ethical argument, but a description of technical capability. If the models improve, do they become ethical to use? The mere slop argument would suggest they do!

It could be an ethical argument, if you form it as a utilitarian argument: they produce more harm than the produce utility. This would require us to admit they produce harm, and start to justify the harm. But I’ve already rejected utilitarianism.

If not careful, the slop argument also attributes the cause of the harm to the generative model. But a model does not, despite the fevered branding of the industry, have agency. It cannot produce something that causes harm by itself. If I go stopper all your sinks and leave the water running in your house, it wasn’t the water that caused you harm.

If the output is poor and you decide to use it in a situation that causes harm, you caused harm. For example, if you submit a thoughtless research paper that wastes reviewer time, it was not the use of the generative model that wasted reviewer time, it was you. If I’m forced to reject garbage generated pull requests, it’s not the generative model’s fault, it’s the fault of whoever setup the model or agent and caused it to go out spewing slop.

This argument might be a good argument for not allowing generative models to be used in some settings. If the balance of probability is that the output will cause harm compared to an alternative, best not to use a generative model. This quickly becomes a technical argument: whether or not the quality of the output can be assured. Perhaps it can’t, so a conservative approach is needed. But this isn’t an ethical argument categorically forbidding the use of generative models.

So I don’t think there is an ethical imperative to not use generative models because some of their output is not of high quality.

The Employment Argument

One argument is see from time to time is: generative models are putting people out of jobs, and therefore you shouldn’t use them. As statement, this is a confused argument in many ways.

For one, an individual’s use of a generative model probably doesn’t put anyone out of a job.

For two, it’s not clear to me that this is necessarily harmful. “Not having a job” is not necessarily harmful, and so this debate quickly devolves into questions about unemployment, about whether new jobs created in place of old jobs, etc. This has nothing to do with an individual’s use of a generative model, and much more to do with the economic systems we’re embedded in, and all the particulars matter.

For three, it’s not obviously true that generative models are putting people out of work. Many CEO’s are claiming that’s why they’re firing people, but there’s plenty of evidence to suggest that’s either a lie to make layoffs more acceptable, or a lie to convince shareholders that the trillions in investments are paying off.

Finally, the generative model didn’t decide to lay anyone off and therefore make their continued existence in a society in which having a job is a necessary precondition to continued survival. Some profit seeking business boy did.

So I don’t really think there’s an ethical imperative to not use generative models based on layoffs.

The Power Argument

There’s only one argument I buy at all, and I don’t see it very often.

Even phrasing this argument requires care: we have to separate the individual use of a generative model from the “AI” industry.

Here, I use “AI”, which I’ve been loath to use until now, very explicitly. “AI” is a brand. It is not a technology. It is a marketing term deployed to convince people that the underlying technology (most recently, generative models) is more capable than it is—that the technology is “intelligent”.

The power argument goes like this: the “AI” industry is accumulating power at the expenses of others, possibly doing harm in the process, possibly using that power to do harm. Therefore one should not use generative models.

This argument… doesn’t follow. Not as is, anyway.

The “AI” industry is concentrating power in the hands of capital, and removing it from labour. The claim of the industry is that tasks previously only able to be done by highly skilled people can—sort of, in some cases—be done by generative models. This would allow someone with a lot of GPUs and a ton data to exercise power over people they previously couldn’t. And the industry definitely wants to use that power, and obviously doesn’t care who they harm in the process.

In fact, all the previous arguments can be augmented into a power argument, and they start to make more sense.

The problem with resource usage isn’t about generative models, but about the “AI” industry. The “AI” industry is spinning up data centre after data centre, without regard for resource use. They sometimes deploy carbon-intensive generators to meet capacity, or consume already scarce water because they have the power to take it from others who need it. These are direct harms that the “AI” industry is engaging in.

The problem with slop is not that one can produce poor quality work. That is not even a new problem. The enshittification process was started long before generative models reached their current capabilities. Long before large language models, others forms of text generation were used to mass produce garbage webpages for small amounts of profit. The problem is that the “AI” industry is wielding an immense amount of power to force “AI” use that lowers the quality of work into new areas. They’re replacing web search with “AI”. They’re replacing customer service with “AI” (chatbots were already common, but this makes them next to trivial to deploy). They’re forcing engineers to use “AI” to generate software. They’re deploying “AI” to make decisions about who to investigate for crimes, who to arrest, who to kill.

The problem with employment isn’t that generative models can take jobs, which I doubt, but that the “AI” industry can replace the output of skilled workers with poor quality output created by generative models. The “AI” industry can replace labour with capital. They’re not even particularly coy about this; some of them have gone on record about this goal. The goal is to gain power over labour, and exercise that power for profit.

Interestingly, sometimes I see the opposite of this argument: people arguing that somehow generative models democratize skilled labour. This is madness. That could, possibly, maybe, be true if these models were tiny and could run on lower powered devices that everyone had access to. Even then, I’m skeptical, as so far I don’t believe they can be used unless you already have the domain skills required to do the work in the first place.

But ignoring my arguments about utility, it’s certainly not true now that generative models give labour power. The only useful ones burn billions of dollars merely to operate, and required almost trillions to reach that state. The only useful models are owned and operated by the “AI” industry. If you want to use them, you have to go to the “AI” industry. That’s not democratization of anything; that’s the “AI” industry having power over you.

So the “AI” industry definitely has power, and they use that power to cause harm. So is there an ethical imperative to not use generative models, based on the power of the “AI” industry?

Well, these harms are not caused by generative models. They’re caused by the “AI” industry, not the underlying technology. They do not go away if you stop using generative models.

But your action to use a generative model may give power to this industry, and thus, cause harm. So let’s consider individual actions.

The Individual Action Problem

The main source of the “AI” industry’s power is economic power. They promise to be able to reduce labour costs, and they need a lot of money to make that a reality (or so their pitch goes).

I’m skeptical that mere use supports the “AI” industry in a monetary sense. The “AI” industry is currently supported by an ungodly amount of investor money and debt, not by the small amount of revenue it brings in. You using a generative model, and even paying a subscription, does not provide support for the power of this industry.

I want to be clear that I’m not rejecting individual action. I think individual action is important. If your action causes harm, no matter how small, I consider that unethical.

I’m saying this particular individual action, using a generative model or even subscribing to a generative model, does not cause harm at all. This is not the ethical problem, because it does not contribute to the power of this industry. If every human on earth boycotted the industry, it would still have as much power as it currently has. In fact, given that most subscriptions appear to cost the “AI” industry money, they might be better off if we boycotted them. We could debate this; I don’t think it’s clear. But let’s accept my premise for a moment.

The money you give the “AI” industry is not the only way to give the “AI” industry power, and taking these other actions causes harm. Using generative models uncritically gives the “AI” industry power, supporting the claims they make and the transfer of power to them. Reporting on generative models and the “AI” industry uncritically gives their claims credibility, giving the “AI” industry power. Enabling their wide-spread deployment, in addition to any harm caused by slop, gives the “AI” industry power over the context in which they were deployed.

In addition to being a hedonist, or perhaps because of it, I’m also an anarchist. I am very against power and hierarchy. I think the main thing they do is cause harm, and we should seek to reduce power and hierarchy as much as possible.

I think there is plenty of evidence that this technology, in the hands of this industry, is causing harm. I don’t think certain individual uses of generative models are unethical, but any action that empowers the “AI” industry is certainly unethical.

So what should you do?

I’m normally pretty reluctant to give direct advice on actions you should take. My standard disclaimer is: all advice is one person’s opinion.

… But we’re several thousand words in and I’ve made my ethical arguments as clear and precise as I can, so strap in buddy while I tell you what to think and how to act.

There is no ethical imperative to not use a generative model; that depends on whether the particular use will cause harm or not.

There is an ethical imperative to deny “AI” (the industry) power.

So how does one deny “AI” power? Well if I knew that, I assure you we wouldn’t be in this mess. But I can tell you some actions I’m taking.

First is education.

I’ve given talks on “prompt engineering” and how it’s not engineering, written these blog posts, and remain engaged in the “AI” discourse despite hating it.

I want people to think clearly about this technology, this industry, and ethics. I want people to understand the technology, which is not magic. I think this is necessary to refute the lies this industry is telling. I think it’s necessary to demonstrate exactly what the technology is and is not capable of. I want people to be able to separate out different concerns and arguments so they can refute lies and nonsense arguments others are telling.

Will this work? I don’t know; maybe.

Second is policy work.

At UBC, I’ve tried to inform policy around the use of generative model. I’m trying to restrict the use of generative models around the university. Some of that is public, some is not. Some has been successful, some has not.

In my own lab, I am in a position of power (ugh), and in that role I try to create sensible policies. I don’t outright forbid the use of generative models; I’m against my own power, and not sure using a generative model is necessarily problematic. I do make clear that the user is responsible for the use of the generative model. Every line of every artifact, if the user cannot stand by it, cannot justify a design decision, cannot explain something, then they, not the “AI”, have failed.

Third is boycotting.

As I said already, I’m not sure that mere use is unethical. It’s certainly not if you have no choice. Even if you have a choice, mere use may not empower the industry.

Still, I, largely, refuse to pay money for this technology. (I spent $10 on various experiments.) I don’t think this has a direct effect, but I think it’s an important line to draw to do what I can to deny “AI” power. I try to avoid technologies and companies and products that adopt “AI” and support this industry.

I think boycotts have indirect effects as well as direct effects. Refusing to engage can cause conversations, it can change minds. That will probably have more effect than denying these companies a few dollars a month.

I have no qualms about using local models. The ones I’ve used are little more than toys. We have some running in the lab that I haven’t tried; maybe I’ll build a machine with a proper GPU and give those a shot.

I do use some industry models; as a faculty member, I can access some pro models for free. I don’t use them very enthusiastically, but I also don’t worry too much about using them. I think using them is necessary to understand them and to educate others. And it marginally costs the industry money when I use them, which is a small bonus.

Fourth is sabotage.

I have deployed lots of “AI” countermeasures, both for training and inference. I don’t know how effective these are, but I deploy them anyway. I should probably go read some activism books, such as How to Blow Up a Pipeline: Learning to Fight in a World on Fire, to find more effective strategies.

I’m not sure any of this will be effective, but I’m ethically obligated to do something.

Addendum

There are many things I wish I’d said differently, and other things I could have considered in this blog post. I’m going to leave it mostly as is, but I will add a few addenda following discussions:

I didn’t discuss one argument that I didn’t see phrased as an ethical argument until after publishing this: the brain rot argument. Mere use of a generative model can harm the user, by limiting stunting their learning. There is scientific evidence that supports this in the context of education. I hadn’t seen it posed as harm, but instead about the effectiveness of the tools for education. However, I find the harm argument compelling, and don’t see why harm against oneself shouldn’t be considered unethical.
One criticism of my “mere use” arguments is that they suppose a “spherical [generative model] user in a vacuum”. This is a valid criticism. In practice, a generative model user may be unable to act as independently of all systems and all harms as I’ve presented, or be unable to correctly judge the quality and therefore harm of their actions. Therefore an ethical user may only exist in theory. That’s very possibly true, and I think there’s plenty of evidence to support it. I think it’s still useful to isolate the pure theory of harm for the purposes of discussion.
In the power argument, mere use may be unethical because mere use may provide e.g. usage analytics that support continued investment and therefore the concentration of power. This is a pretty compelling argument, and is probably related to why I instinctively boycott these things.

There are certainly other arguments I haven’t considered, but as I said in the beginning, my conclusion is not the important part of this post.

Against Vibes: When is a Generative Model Useful

2026-03-05T23:58:20Z

Let’s suppose I wanted to answer a question: is the tool X useful for the task Y. If I were scientific about this, I would analyze the properties of tool X and develop a model, and the task Y and the requirements for it and develop a model, and I would use my models to predict the behaviour of tool X in the context of task Y. “Can I use timber instead of stainless steel as a support beam for this structure?” “Will this acid be an appropriate solvent for this reaction?” “Will this programming language provide these real-time guarantees?”

The discourse on generative models is not like this. Instead, you get claims like “software engineering is dead”, and attempts to shove generative models into literally everything without a thought. Search? Generative models. Code completion? Generative models. Summarization? Generative models. Voice to text? Generative models. Stock images? Generative models.

Any attempt to criticize this tends to go in circles and/or have people arguing past each other. Is a generative model useful for internet search? Well, look, it produces text that is plausibly related to the input prompt, so. So… what? That doesn’t answer the question. But the newest models are so much better! Better at.. what?

I was upset about this when it was being called “prompt engineering”, and found no sign of engineering, but instead a series of vibes about how to phrase a prompt in a particular version of a particular model, which sometimes produced output that is plausibly related to the input prompt and therefore plausibly close to what you might have intended. I’m upset now when people are making claims that agents are so useful, but can’t tell me when or why or how they’re useful beyond vibes about feeling more productive (vibes that have been refuted by real science contrasting objective measure of productivity vs. subjective reports), or examples of having produced a lot of plausible output.

(Okay, there are some researchers doing actual science and writing papers; I’m talking about the arguments being made as this stuff is integrating into schools, workplaces, etc.)

I want to know when generative models are useful. I don’t want to feel like they’re useful, that’s just a vibe. I’ve been a generative model skeptic basically from the beginning. I could not convince myself that generative models were useful. But I was also skeptical of my own subjective experience. I could imagine that a model capable of produce code from natural language would be useful, in some use cases that I had not found. I imagine there must be a model of when a generative model X is useful for task Y.

In this post, I’m not addressing ethical, political, or social questions. Those questions are important, and I want to address them separately from what the technology is capable of. Just for context: I think the widespread deployment of this technology is deeply problematic and irresponsible. I think further investment in it at its current scale is an almost criminal level of fiduciary negligence and will cause economic harm. I think the ethics of all of this are deeply troubling.

But for now, I just want to know what they’re technically capable of.

A model of generative model utility

I think the usefulness of a generative model is a function of three things:

What is the cost of encoding a generative task in a prompt vs. directly producing the artifact? This is a function of the task, the model, and the user.
What is the cost of verifying the generated artifact meets requirements vs. a directly produced artifact? This is mostly a function of the task and the user, but also the generative model.
How much is the task dependent on the artifact vs. the process? This is a function of the task.

Each of these touch on things many others have said, but I think all three considered simultaneously are important. They make it possible to be scientific in an argument about the use of generative models.

If you want to claim a new model is “more useful”, you must specify all of these variables. You must specify a class of tasks, and demonstrate the cost of encoding for some set of users is lower than directly producing the artifacts of tasks; or that perhaps encoding is higher, but that verifying design requirements is lower.

More importantly, if I want to predict whether a generative model will be useful, I have a model to work with.

My model predicts that usefulness of a generative model may decrease as task complexity increases. Generative models are probabilistic: the output will be less likely to satisfy complex requirements, particularly if those requirements differ from common patterns in the training data, or worse, subtly different from common patterns. Verifying complex requirements is also hard, and harder than having a human following good engineering processes that lead to more easily verified outputs.

On the other hand, generative models should be useful when directly creating the artifact is hard for the user, but verifying the artifact is trivial. This could be the case for artifacts that require cross-referencing extremely specific information that is time consuming for a user to do, but once done, is trivial to check. It could also be the case for generative models integrated into formal verification systems with extremely reliable and highly automated verification, where no knowledge of the artifact being generated is necessary. But in general, it is unlikely to be the case for a novice in some domain trying to generate a complex artifact, since the user will not have the expertise to ensure the output meets requirements. This predicts there will still be a need for users of generative models to have domain expertise.

The model also predict that generative models are essentially useless for tasks that are highly process dependent, since all a generative model can do is produce an artifact by a blackbox process.

1. Relative Encoding Cost

A lot of arguments in favour of the usefulness of generative models make arguments about, in essence, the relative encoding cost. For a generative model to be useful, the total encoding cost must be lower than the total cost of directly producing the artifact.

The total encoding cost includes all the work that goes in to writing a prompt, and all of the compute required to run the prompt. If the task is simple to express in a prompt, the total encoding cost is low. If the task is both simple to express in a prompt, and tedious or difficult to produce directly, the relative encoding cost is low. As models get more capable, more complex prompts can be easily expressed: more semantically dense prompts can be used, referencing more information from the training data. An agent capable of refining or retrying a task after an initial prompt might succeed at a complex task after a single simple prompt. However, both of these also increase the compute cost of the prompt, sometimes substantially, driving up the total encoding cost. More “capable” models may have a higher probability of producing correct output, reducing costs reprompting with more information (“prompt engineering”), and possibly reducing verification costs.

Moreover, as a user gets more capable, they may be able to complete the tasks directly much faster than they can prompt a model to do it, driving up the relative encoding cost.

One may argue that the newest models are “more powerful” or “actually intelligent” or whatever. But these are unscientific claims.

The scientific version of these claims is “the total encoding cost (for some class of tasks) is lower than previous models”. Phrased this way, it’s clear this still doesn’t mean the new models are useful.

For most of my tasks, I think the relative encoding cost has been high. Many of my software engineering tasks are constructing small, semantically dense programs, with very specific design requirements, in a language much more concise than English that I can write fluently. I can systematically design and implement such software faster than I can encode the specification into a prompt for a generative model.

As one example, I tried using Claude Opus 4.6 to generate a program that would interpret a custom DSL I use for typesetting grammars, and generate Haskell type definitions. After 8 hours of prompting, several million tokens, the code it generated was still absolutely useless. It passed the tests I had prompted it on, but just looking at the code, one could easily identify type errors and logic that tried to special case specific identifiers from the tests. The logic for sanitizing identifiers was a mess, and would occasionally generate empty strings. A correct implementation would take me 300—400 line of code to write, which I can certainly write in less than 8 hours.

However, not true of all tasks are writing semantically dense code with very tight design requirements. For example, I was recently trying to install a package whose name I forgot. I prompted the model to “install that x11 fake gui thing”, a trivial prompt. Actually completing the task myself would have required a lot of tedious work, with lots of accidental complexity. I would have needed to search the internet to identify the name of this software, cross-reference that with the distribution of the operating system I was running and the name used by its package manager, possibly cross-reference the installation command for this particular package manager, and then write and execute a shell script to perform the install. I was able to use the agent to do all of this with an extremely easy to write prompt. This task had a very low relative encoding cost.

2. Relative Verification Cost

Some arguments about generative models focus on verification: “formal verification will become more important as more code is generated”.

I think these arguments are also unscientific. Verification is also not magic. Software designed in one way may be easier to verify than in another way. A user who carefully designed and implemented the software may be able to more easily verify it than when dropped into a fully generated code base.

All of this depends on the task, the user, and what the model is capable of generating. For my prompt to install a package, verification is trivial. I will recognize the right command when I see it, and it’s one line long.

Relative verification cost also goes up with the size of the task. If you’re generating a 1 line script, no problem. If you’re trying to generate a very large artifact, you’re going to get bored validating every command, every edit. You’re not going to be able to check every line of code that was generated. You’re going to need some other approach to verifying the output, increasing cost.

Relative verification cost also depends on the user. If I’m prompting a model to produce Racket, a language I am very fluent in, I can quickly evaluate the design and implementation of the generated code. If I tried to prompt a model to produce C, I’d be far better off just writing the C myself, following a systematic approach that would result in safe C. And then running it in a sandbox. After running some sanitizers on it.

Relative verification cost somewhat depends on the capabilities of the model, too. Some of the early models I experimented with produced trash code. Not merely bad code with bad design, but errors so basic I wouldn’t think to look for them: it would produce Racket with mismatched parenthesis, references to functions that didn’t exist, etc. Those are easy enough to detect by running the compiler, but what about the ones that aren’t so easy to detect?

One key part of this relative verification cost is that generative models produce plausible output. It’s not accurate to say a model produces “correct” or “incorrect” output, or “makes mistakes”. It does exactly what it’s designed to do: produce output that is statistically related to the input prompt, in some way. That doesn’t mean “statistically correct”, just “statistically related”. All output is correct, in the sense that all it’s suppose to be is a point in the distribution of things related to the prompt. Maybe you produce C code with memory errors most of the time, but most C code has memory errors. Maybe you mostly produce correct bash scripts for installing packages, because most bash scripts for installing packages on the internet are correct.

Plausibility of generative models greatly increases the relative verification cost, since the output is essentially optimized to be close to correct. I’d predict that relative verification cost could go up as the models get more complex. The class of errors we’re likely to find in generated code will be very different than the class of errors we’re used to looking for in human generated code: generated code will have subtle errors. As the models get more capable, you might be more likely to trust the output, and less likely to spot these subtle errors. This cost can be reduced by formal methods, but formal methods aren’t necessarily cheap. You might be better off with an engineer following a design process.

For some tasks, verifying the output may be impossible, or at least impossible without redoing exactly the work you were trying to use a generative model for. I think internet search is a good example of this. A generative model generated response to an internet search query that you did not know the answer too is essentially unverifable, unless you go and search for credible sources to verify the summary. At which point, the generative model did entirely wasted work.

3. Artifact vs. Process

Some tasks aren’t about the output. Or maybe they aren’t just about the output, and require the output be created following a specific process.

Easy examples of this are in education. I don’t need students to implement factorial for the one billionth time because I need an implementation of factorial. They implement factorial because going through the process creates knowledge in their head. Writing code is a fundamentally different process than reading code, in the same way that writing this blog post is a fundamentally different process than reading it.

This blog post is an example of a process-driven task. I’m writing this post. My hands are typing the words that appear in this post. They are not merely typing prompts that cause a generative model to generate plausibly-related words. That’s because I’m not trying to create a blog post. I’m trying to create knowledge, within myself and then within others. Writing this post is me thinking through all the details.

Process-driven tasks also come up in engineering. Some engineering requires specific processes are followed, because if the processes are followed, the end result will satisfy certain properties that may be difficult or impossible to verify just looking at the artifact. A large fruit company, for example, might forbid engineers from contributing to open source projects as part of the process by which they engineer software in order to mitigate intellectual property risks. There is no way, just looking at the software the engineer writes, to guarantee freedom from those risks.

The process argument against generative models comes up a lot. The argument goes something like “X is about human communication, or creativity, so generative models cannot be used to create Y”. And I really sympathize with this argument, because I think far too much is produced by ignoring the process.

But there are certainly tasks for which I really only* care about the output. My shell script example is one: I don’t care how the package gets installed; I care that it’s installed. (* Well, assuming the output wasn’t produced through some truly problematic process, which, well… but that’s a future post.)

The same can be true of writing. I am writing this post manually, because the process matters, but some writing is functional. For example, I used a generative model to draft a policy document. Policy documents don’t have much creative structure to them; they express a set of rules. I’m still reading and redrafting the document, since I need the particulars to better suit me, but it was useful to start from a generated draft, in the same way I might start from a template.

But even when the output is boring and easily verified, the process may be important. Junior engineers might write lots of boring, easily verified code. It might be extremely cheap the replace them with agents. But junior engineers writing that code are going through a process by which they gain experience, knowledge, and skills. Generative models can replace their output, but nothing can replace that process.

For almost all software I write, I do care about the process. I’m typically designing software as part of research, and me doing the design and implementation work creates knowledge that I will then share. The software isn’t the important output, or not the only important output. I think this is another big reason I haven’t found these things useful, and why it’s been such a struggle to figure out how they could possibly be useful.

We just need people who can make a computer produce useful work

So when is a generative model useful? Just when the (1) relative cost of encoding the work in a prompt is low (compared to doing the work some other way); (2) and/or relative cost of verifying the output satisfies requirements is low; (3) and the process used to complete the work doesn’t matter. To judge all of this accurately, the user of the model needs to know quite a lot about the work being done, about verifying design requirements in the domain, and about working with generative models and/or the model in question.

Navigating these trade-offs is engineering. If you’re navigating those trade-offs to produce software, you’re doing software engineering. If you’re not considering these trade-offs, you’re just going on vibes and what you produce will be something between accidentally useful and extremely harmful.

These trade-offs aren’t unique to generative models, but one thing is: they’ve made it incredibly cheap to produce an immense amount of output that is plausibly described by a natural language description. But plausible doesn’t mean useful, and there’s nothing in generative models that could ever guarantee useful output. As the models get more sophisticated, the complexity of the output and the prompts are getting more sophisticated. That’s not necessarily more useful. As that complexity goes up, so do the costs: of compute, of verification, and of relying on output over process.

I understand the temptation of these tools. Sometimes useful work is incredibly complex and frustrating to do. Writing software, running scripts, and organizing all my notes can be very tedious. Sometimes that is accidental complexity, but much of the time it is essential. It is very easy to use a generative model produce output. I don’t think it’s very easy to use them produce useful output.

Are the ACM's profits supporting its mission yet?

2025-10-03T22:08:18Z

Last year, as UBC got involved in ACM Open negotiations, I got curious about ACM financials. As I dug into them, I didn’t like what I saw, and wrote a blog post and then a CACM Opinion piece. The ACM honoured me with a response article, which includes lots more data and a contrary perspective on the ACM’s $250,000,000 in assets and $10,000,000/yr in profit. The response also made an interesting claim:

In terms of DL subscription revenue alone, the ACM is projecting a loss of $6M to $8M in 2026 when the DL goes fully open. The five-year projection is for a loss of $25M to $30M.

I was very interested in this claim. In making it, they have activated the trap card of falsifiability.

I’m quite happy that the ACM is moving to an open access model, and want that to be financially viable. But, as I said last year,

I do not think it is in the public or professional interest, nor does it advance art, science, engineering, and so on, to charge unnecessarily high publication and conference fees, taken out of public, research, and educational funding, and to hold that surplus income as an increasingly large pile of assets.

So I was keen to pay attention to this claim about the need for that surplus, and its use in supporting ACM Open.

Well I’ve got to reading the new ACM 2023 and 2024 publication finance report, and the new FY2023 IRS data, and attention I am paying. So… is the ACM losing money in the switch to ACM Open, and is its hundreds of millions in profit over the past decades helping it achieve its mission?

TLDR: Na. The ACM remains profitable and its net assets have continued to grow by tens of millions of dollars a year. And IMO, some of their rhetoric is even more worrying than that.

First, some context.

As in my prior article, I will rely heavily on the IRS 990 filings. Since my previous article, FY2023 has become available. This data provides a good overview of the considerable income from all sources, such as conferences, gifts, ads, investments, and membership fees, and all expenses. However, the IRS reporting uses a different financial year and different categorization of income and expenses than the ACM uses internally.

ACM have also recently published the 2023 and 2024 finance report, which I will pull from. These data are incomplete: they only report the finances for the publishing arm of the business, sorry, enterprise, sorry, non-profit mission that is the ACM. However, it includes more fine-grained information than the IRS data.

The differences between these two sources could result in apparent contradictions that are in fact totally benign. I won’t dwell on this. Both sources tell interesting stories, both separately, and together.

Okay! Let’s look at income.

Notable facts and figures about income from the ACM finance report:

ACM DL subscriptions dropped from $11,500,000 to $3,900,000, representing a loss of revenue of about ($7,600,000). However, this is totally expected, since the ACM DL subscriptions are being phased out in favour of ACM Open.
ACM Open revenue climbed from $7,700,000 to $15,300,000, representing an increase in revenue of about… $7,600,000. Fancy that.
In total, publication revenue increased $200,000 from 2023 to 2024, from about $25,000,000 to $25,200,000.
In total, revenue has increased about 1% per year from 2019—2024.

So, ACM Open doesn’t seem to be harming revenue much at all. And, as a result, science has become much more accessible! Well done ACM! That’s how you serve the public and member interests!

Revenue growth at 1% seems problematic, since it’s below inflation, and below the increase in expenses. But, that’s partly by design: the ACM has frozen prices during the transition to ACM Open. So the fact that revenue is growing despite the transition to ACM Open cannibalizing other sources of revenue, providing free services to previously paying customers, and not increasing prices to keep up with inflation, suggests that actually ACM Open is… overpriced?

The finance report couches this in other term that I completely disagree with. They start with the claim that:

it’s reasonable to ask why income growth has been so slow?

And go on to assure the reader that these restrictions on income growth are temporary and, not to fear, income will begin to grow again soon!

But you know what: no, no it’s not reasonable to ask why income growth has been slow. That was the entire point of my original article. The mission of the ACM is not to grow income—to make a profit—but to serve the public and professional interest. The ACM should not be aiming to make a profit, and yet it’s clear that they both DO make a massive profit, and believe it is part of their mission to do so.

Let’s zoom out and look at the IRS data. In FY2022, the ACM made $12,000,000 in profit, while in FY2023 they made $380,000 (which is an interesting outlier I’ll return to). Their total assets are up from to $211,500,000 to $264,000,000; I use total rather than net, because a major liability is deferred revenue, which is not a liability in the same way as other debts.

So the ACM does not appear to be on track for a $25,000,000 to $30,000,000 loss, and is still certainly not using their hoard to support its mission. ACM Open does not seem to be harming revenue, or profit much, even with its temporarily limited price increases.

But.. what about that drop to $380,000 in profit, from an average of $11,000,000 per year for many years?

Let’s look at that more carefully. This drop seems to come from two places:

Revenue, but importantly not program service revenue (i.e., from ACM Open) fell a bit.
Expenses, but not publication expenses, increased a lot.

As we’ve seen, revenue from DL subscription and ACM Open balanced out. All program service revenue (which includes conferences, publications, memberships, and advertising) increased from $69,000,000 in FY2022 to $74,000,000; it’s way up. Membership fees and advertising revenue is down, a bit. Conference revenue is up, from $39,000,000 to $43,200,000, although so are conference expenses. Contributions, which includes gifts and grants, have remained essentially the same at $8,000,000 in FY2021, $10,000,000 in FY2022, and $8,000,000 in FY2023. Investment income is down a lot, but possibily due to an outlier in 2022. Between investment income and investment sales, it was about $7,000,000 in FY2021, $12,000,000 in FY2022, and only $5,000,000 in FY2023. So income is pretty stable, but down.

Now expenses.

Again, I’ll start with the publications finance report. The expenses section is weird, but I’ll start with the reported facts and figures.

Notable facts and figures about expenses:

In total, publication expenses fell from $29,200,000 to $28,800,000 from 2023 to 2024, saving $400,000.
Digital library expenses, particularly infrastructure expenses, increased from $6,800,000 to $7,900,000, by about $1,100,000.
Anything related to print publication fell, enough to offset all that DL expense and then some.
In total, expenses rose about 4.4% per year from 2019 to 2024.

Notably, publication expenses are still greater than publication revenue. This has been the case for some time, but the profits from conferences offsets that difference. Recall the ACM is still reporting a profit, and a substantial growth in assets.

The expenses also report some… bizarre… claims. The two claims are:

overall expenses have continued to increase, with the most significant increase of 27.91% occurring between 2023 and 2024, when the program grew from 29,691 articles in 2023 to 37,978 articles in 2024.

Over the longer period from 2019–2024, expenses grew at a rate of 13.66% per year.

These claims are.. nonsense. The expenses clearly didn’t grow 27.91% from 2023 to 2024, and nor by 13.66% per year, which obviously contradicts the earlier (correct) claim that they grew by about 4.4% per year. I believe they’re meant to say that the number of articles published increased by 27.91% from 2023 to 2024, and 13.66% per year. That at least matches the number of articles reported in the sentence. I can only assume the claims are typos. I’ve reached out to the director of publications, who has said my interpretation is likely correct and that they will fact-check this, but it has been some weeks and I want to finish writing this post.

The IRS data on expenses basically supports the data in the finance report. A nontrivial amount of the increases was on DL expenses. This is reflected in the IRS data as “information technology” expenses, which are up to $7,000,000, from $5,700,000 in FY2022. Publication also does seem to be getting slightly more expensive. Publication production and services at up to $6,700,000, from $5,300,000. But, that is not nearly enough to make up for the $12,000,000 drop in profit.

Conferences appear to be a big expense. Conference expenses are up to $41,700,000, from $36,700,000 But conference revenue has grown to match, and still subsidizes other parts of the business.

Lots of little things went up: employee salaries, travel expenses, occupancy, amounts paid to independent contractors such as Conference Publishing. So, yeah expenses are going up, and without increasing revenue, and with a bad year for investments, the ACM didn’t make quite as much in profit as usual. (Although investments still appreciated by $10,000,000, that’s not counted as revenue since the gain hasn’t been realized yet.)

What’s more interesting is to try to understand WHY these expenses are going up. Sure occupany, employee salaries, conferences—those are a bit out of the ACM’s control. The ACM believes it can reduce publication expesnes:

ACM had expenses of $28,854,059, and while we anticipate being able to reduce this figure over the next few years

So hopefully those are growing pains.

But one place where expenses are increasing is very telling.

In its own words, the ACM believes income growth is part of its mission, and is spending money to increase future revenue. A non-trivial amount of the increase in expenses is on the digital library, but not for anything that benefits the public or members interests: on product differentiation to grow profit.

ACM has invested heavily … and embarked on a large-scale initiative to launch both Basic and Premium versions of the DL platform with robust features, functionality, and value-added content in 2026 [including] advanced search tools, such as AI assisted search; and research insights, such as usage metrics, citation trends, institutional and author profile pages, and altmetrics.

For any researcher, student, practitioner, or educator working in this domain, access to the Premium version of the ACM Digital Library should be considered essential.

ACM will work hard to identify new value-added products and services for the researcher, practitioner, and educator communities,

This is absurd and insulting. We don’t need to spend money on AI assisted search, supporting an ethically dubious industry and bubble to get slop full of bullshit (in the technical sense) in our search engine, especially if we need to control costs. That approach has made search engines increasingly useless. We don’t need more stupid, useless metrics for bureaucrats to optimize at the expenses of real value. All of this is make work; all of this is bullshit jobs.

Sure, ACM Open prices might need to go up over time to keep up with costs, maintain the digital library, etc. I understand the need to have a robust network infrastructure to serve all our papers and archive them indefinitely. But some these costs are self inflicted, and the ACM has openly declared a goal of profit seeking product development that has nothing to do with its mission.

Sooo … is the ACM losing money on ACM Open? No. Revenue loss from DL subscriptions are matched by ACM Open revenue increases. The ACM still appears to be extremely profitable overall. Unless things have gone completely off the rails since FY2023, the ACM is still profitable and now has a hoard of ~$280,000,000 (projected). If you consider publications in isolation, yes, ACM Open is losing money, but is more efficient than DL subscriptions—it lost about ($3,600,000) in 2024, compared to a loss of ($4,200,000) in 2023. Personally, I think it’s a little disingenuinous to completely ignore conference revenue when considering publications; the profit from conferences almost entirely covers the loss on publications.

Is the ACM spending down its hoard? Obviously not. It continues to grow, and at essentially the same pace as before.

Is the ACM using its profit to support its mission? Well. I’ll give them one thing: with the price freeze in place despite increasing costs, the ACM has cut into its profit, mostly to support its mission. But it also spent a huge chunk on for-profit behaviour, developing new nonsense products and setting an explicit goal of increasing profits. The ACM is profiting, and making it a goal to profit, at the expense of its members and/or the public it is meant to serve.

What's higher-order about so-called higher-order references?

2025-06-02T21:56:43Z

A reviewer, in one of those off-hand little low-level comments they leave, which one is sometimes tempted to ignore as one works on a revision because they’re not strictly actionable, has absolutely nerd-sniped me.

What does “higher-order” in “higher-order references” mean? (References as in Reynolds 1970.)

We appear to have inherited the phrase “higher-order” from higher-order logic, which it refers to the union of first-order logic, and second-order logic, and third-order logic, and so on. Here, “order” refers to what quantifiers can quantify over. For example, zeroth-order logic has no quantifiers. First-order logic can quantify individuals. Second-order logic can quantify over things that quantify over individuals (sets). Etc.

So what is a higher-order function? Well, a function that can quantify over functions that can quantify over functions, etc. Really, we could perhaps generalize to higher-order values to make a closer analogy to higher-order logic. A zeroth-order value cannot quantify over other values—values such as naturals and booleans. These are sometimes called ground values. A function cannot be a zeroth-order value, since it quantifies over other values. A first-order value can quantify over zeroth-order values. A fairly restricted function would be a first-order value. A second-order value could quantify over values that quantify over values, so for example a function of type (A -> B) -> C, where A, B, and C are ground values, would be second order.

So far all this makes sense.

What is a higher-order reference?

Well… that is a surprisingly complicated question.

Of late, “higher-order reference” seems to mean “reference to a higher-order, effectful function”. The phrase appears in this sense at least as early as 1998 (Abramsky et al.), in which Abramsky et al. use “higher-order reference” to refer to references that contain higher-order functions that can, when called, modify the heap. But this use of “higher-order” is at odds with most other uses of “higher-order”. It’s not used to mean that the reference merely quantifies over other references. Instead, it’s used to mean references that quantify over higher-order, effectful functions, a different class of values. That is, this use of “higher-order” refers not to the order of the reference, but the order of the values the reference contains. If we were to use “higher-order reference” more consistently with “higher-order logic”, it should mean references to references to references etc. to values.

The phrase “general references” avoids the problem, but does not solve it. This phrase is often used, including by op. cit., to include other forms of higher quantification in references, such as cyclic references and references to existential types, etc., as well as references to higher-order, effectful functions. To quote:

By general references we mean references which can store not only values of ground types (integers, booleans, etc.) but also those of higher types (procedures, higher-order functions, or references themselves).

Ahmed (2004) uses similar phrasing:

general references — that is, mutable cells that can hold values of any closed type including other references, functions, recursive types, and impredicative quantified types.

The emphasis is on their generality, which distracts from a key semantic issues (higher-order, effectful functions in particular are responsible for interesting semantics questions).

The situation is worse if we look at the phrase “first-order references”, which also appears in Abramsky et al.. This appears to refer to references to ground values, i.e., references to zeroth-order values. So a first-order reference is indeed a first-order value, in that it quantifies over zeroth-order values. But that means “first-order” in “first-order reference” is used in a difference sense than “higher-order” in “higher-order references”: it refers to the order of quantification of the reference itself, and not to the order of the quantification of the contents of the reference.

Even more frustratingly, it seems the phrase “higher-order reference” used to use “higher-order” in the original sense. Janssen et al. (1977) use “higher order reference” to refer to references that quantify over references! Readings over some of the related papers around that time, it seems the semantics of such references were quite interesting. It’s unclear to me when exactly the phrase took on the new sense of “higher-order”.

So how do I concisely, and pedantically, refer to those references that can contain higher-order (effectful) functions (but not necessarily other non-first-order values)? My student Sean Bocirnea proposed “higher-order value references” (HOVR; pronounced “hoover”). Perhaps we’d use “higher-order function reference” (HOFR; “hoofer”) to speak of those only required to contain higher-order functions. These are less verbose than any alterantive I’ve come up with, but have bad mouth-feel.

Given the phrase has had a new sense for at least almost 30-years old at this point, I’m a a little reluctant to be inventing new phrases. Perhaps I should just leave it at this, with this note serving as a historical marker of this accident of language. But I’m also insufferably pedantic, and the phrase does have two meanings, so perhaps I should create a new phrase.

Academic freedom, freedom of speech, and politics

2025-04-10T17:19:10Z

Recently, there has been an attack on academic freedom and free speech at UBC, by fringe right-wing extremists, under the guise of protecting free speech. They recently escalated this attack from bullshit internal political wrangling (which I’ve been helping fight), to a BC Supreme Court Petition: Petition-VLC-S-S–252602-filed–7Apr2025.pdf

I’m so tired. So angry. This is such bullshit.

Let’s talk about academic freedom and freedom of speech.

This whole petition is nonsense. It rests on the false claims that:

UBC is forcing faculty to make take certain political positions.
UBC is requiring faculty to express certain political speech.
UBC is required to be apolitical, which it is not if it makes statements on political topics.

All of this is nonsense.

UBC is not limiting the speech of faculty, nor requiring faculty to make any speech and this attempt to ban speech under the guise of free speech and being apolitical is the same kind of attempt to destroy institutions that’s happening in the US. But, if you intentionally mislead people and conflate different categories, it could look like UBC is being political. So let’s deconstruct some categories.

The petition conflates “UBC” and its administration with the faculty that make up the university, and conflates “speaking on topics related to politics” with “being political”. It is vital that UBC faculty maintain their academic freedom, including to speak on political topics. It is vital that UBC faculty be able to state facts about political topics; that is not being political.

Unrestricted speaking on political topics is vital to academic freedom. All this petition does is try to limit faculty from one kind of speech, thereby benefiting one perspective (and faculty that have their perspective) over another. Merely stating facts is not political, even if those facts are related to politics. Even sharing expert analysis and opinion about political topics is not “being political”. The intention of a university being “apolitical” is for the university to serve the public, regardless of their political perspective, and for its faculty to be protected from political whims, regardless of their political perspective.

What is the university and why do we think it needs to be apolitical?

We faculty need to, need to be trusted to, provide expert analysis, opinion, and education, to everyone, equally—regardless of political stance. The faculty need to be able to speak freely, even and especially on topics that are politically sensitive, to do that. That means they need protection from political whims in the exercise of their academic mission. That doesn’t mean not speaking about politics. It means exactly the opposite: enabling and protecting faculty and groups of faculty to speak about political topics.

So being “apolitical” means to be protected to work on, for, and with anyone regardless of politics. It does not mean “not touching politics”.

Consider a particular policy, say introducing a wealth tax. It’s critical that faculty be able to study, and argue the advantages and disadvantages of a wealth tax. Or, in the exercise of their expert opinion, to argue for a wealth tax as a good policy, or argue against it as a bad policy.

Being “apolitical” does mean that UBC, as an institution, cannot unreasonably restrict the work of the faculty that make up UBC based on political considerations. We can’t have a university administrator telling faculty what to study or support a particular policy. We can’t have university say, as a whole, it is in support of such a policy, over the views of the individual faculty. We can’t have the university come out and say “We’re only providing expert opinions to the liberal party”. That would be “the university being political”.

But that is a straw man that just isn’t happening, and by attempting to restrict speech related to politics to prevent that straw man, you’re actually restricting the speech of individual faculty in the exercise of their academic mission.

This petition is making the same mistake as the above strawman (I’d guess, intentionally).

It is not political to state that “UBC exists on the unceded ancestral territory of the first nations”. That’s just a fact. It does.

It might be political for UBC to come out and say “and we should give it back”. Probably not all faculty agree with that, such as those putting forth this petition.

However, any individual faculty must be protected in saying “and we should give it back”, because they may want to make an academic, political, ethical, sociological, or economic argument about this topic, and conclude that this would be a good policy. Unfortunately, by the same token, individual faculty must also be protected if they want to make the argument that “and it’s super good that we killed a bunch of people and took their land”. That’s a pretty unpopular argument, and I think it would be hard to make, and while you probably shouldn’t lose your academic job if you’re making that argument academically, everyone is in their right to tell you you’re a bad person for making it.

It is also not political to say “We condemn killing children, bombing hospitals, murdering journalists. This meets all the definitions of a genocide, and is bad”. It might be political to for UBC, as an institution, to then suggest we should break off ties with Israel. But individual faculty must be protected in making that argument. It would certainly be a restriction of academic freedom and free speech to say faculty cannot say that, or make that argument. Unfortunately, by the same token, individual faculty must be protected in making an academic argument that Israel is within it’s right to wage total war. Although, that would be a tough argument, given the clear violations of international law. And everyone would be well within their rights to tell you you’re a bad person for making such an argument.

I’m being a little extreme in the above examples; academic freedom isn’t that absolute. Governments can, and do, reasonably restrict faculty from certain kind of speech, and impose other reasonable limits on rights to limit harm. So maybe actually a faculty can’t go around arguing “genocide is good”, because they might incite people to violence and cause a lot of harm. But at least in principle, even distasteful arguments on political topics need to be protected, and at the very least it’s not the university’s role to restrict such rights, but a government’s role. But it would certainly be unreasonable to restrict faculty from making the argument that genocide is bad. What would be the harm; inciting people to stop doing genocide?

Similarly, it might be that a government imposes a reasonable restriction to advance some good. If land acknowledgement statements were political, and I’m not convinced they are, they could be a reasonable restriction of speech in order to undo a harm or advance a common good. If this was a policy set by a government, I wouldn’t have much to say about it. I would still argue that a university shouldn’t set such a policy, and faculty should be protected if they want to argue against that policy (even as they are bound by it).

This attempt to ban certain political speech at the university is a blatant attempt to reduce academic freedom and restrict free speech, and is itself being political. By preventing faculty from expressing facts and opinions on political topics, you benefit exactly one side: the side currently in power that is being spoken out against. Doing that is political; it makes a policy decision. It shifts political power. Exercising academic freedom to speak on political topics is different from “being political”.

If we had a policy that says “UBC can’t support Palestine”, does that mean the institution can’t, or are you restricting the individual faculty? Given that UBC, as an institution, isn’t doing this, this petition can only have the effect of restricting individual faculty. This petition is political, and is trying to restrict academic freedom and free speech.

The Structure and Interpretation of Computer Science Academic Metrics

2025-04-02T20:39:11Z

I’ve been unhappy with metrics lately, watching academics (of all people) apply metrics without interpretation. I’m losing my mind over it.

This was free written in one sitting don’t @ me.

Publication Count

This probably started when my department introduced a new policy explicitly measuring publication count. They wanted an objective, transparent metric of research productivity. They decided (well, some of them decided, and pushed through a policy against all department norms) that the metric of research productivity would be (drum roll please): publication count. At least 1 paper per year (ish) would be required.

As if “1 paper per year” is a meaningful metric. As if “1 paper” is a thing that exists. We all know there is no consistent interpretation of “1 paper”. There was much discussion on this point. Everyone admitted that “1 paper” varies greatly from field to field. Some papers are smaller, some are large, some take many years to do the research, write, polish, and some are small results easily accepted in small venues. But surely, “1 paper” is the bare minimum anyone could possibly be expected to publish in any field. So the policy was codified.

Publication count has long been a proxy for research productivity, and while it is objective and transparent, it’s also meaningless.

Number of physical chairs in a department is also objective and transparent, and almost equally meaningless. Surely most chairs in a department are assigned to people doing research? They need a place to sit in order to do research. So as the number of physical chairs increase, the research productivity increases, because more researchers are sitting in chairs doing research. Sure, there are differences between chairs. Some are rolly chairs, some have lumbar support, some are mere stools. Some, of course, are assigned to admin staff, and really only indirectly support research productivity. It’s not a perfect metric, but it’s objective and transparent!

I’m generally skeptical of metrics. “When a measure becomes a target, it ceases to be a good measure.” — Goodhart’s Law. Merely the act of measuring changes peoples’ actions. It provides an incentive to game the metric. You can use evolutionary game theory to show that introducing a metric as a fitness function will destroy any other variable, say, quality. If only the fittest survive, then anyone who better optimizes the fitness function wins and survives. When the fitness function is “number of publications”, and if this is not identical to “quality” (it’s not; it takes longer to polish a paper than to merely publish one), then trying to maintain “quality” is a losing strategy. Any rational actor seeking to survive will change their strategy, avoid “quality”, and optimize “quantity”. Any irrational actor, seeking to maintain “quality” over “quantity”, will lose and not survive.

“1 paper” is not a measure of research productivity. It is a measure of how many papers one has written, which is loosely correlated with, but not the same as, research productivity. It’s fairly easy to write papers which no research content. There are lots of venues that will accept a tutorial paper, or a neat idea paper, or controversial opinion, or a talk abstract. I’ve written some of these; it’s not bad to write them. But they’re not necessarily the same as research productivity. I would not equate my ACM Opinion article (a paper, with a DOI) with a POPL paper. That would be absurd. I wrote that as a blog post late one night after getting mad about ACM fees. Probably my “lowest tier” publication is a paper at Scheme Workshop. Scheme Workshop appears to be slowly dying, doesn’t have a formal proceedings (most of the time?), gets few submissions and the accept rate is almost 100%. But that paper involved a year of writing code, documenting, deploying to hundreds of users, testing, writing 24 pages of technical explanation, and preparing and delivering a 30 minute talk. That’s some research output. I’d have been smarter, more fit, to submit a neat little toy in Scheme that took no time to write about.

How could academics, trained in studying and understanding data in minute details, so naively apply metrics this way? Don’t tell me; I know the answer. They wanted to make decisions more easily, and they found an objective, transparent metrics to do that.

Emery Berger’s Ranking

This is literally the reason http://emerybergersrankings.org was was created. To provide an objective, transparent metric to aid grad student in making decisions about which grad school to go to.

I also hate Emery Berger’s Ranking. The goal is to provide “a metrics-based ranking of computer science institutions”. Ranking of what? Metric that measures what? It measures the output at a completely arbitrary list of conferences divided by author count per paper. What does that tell you? Why is is the school that is Number 1 in Arbitrary Conference Set Papers Divided By Author Count the “best” school? The one you want to spend ~6 years of your life studying at? At best, Emery Berger’s Ranking tells you that going to the most highly Ranked school will probably result in you producing some conference publications, likely with fewer coauthors than at other institutions? If that’s your goal, this is a great metric! Otherwise, it’s a bad metric. Honestly, US News and World report is a better metric for choosing a grad school, as it takes into account things like how nice the campus is, which greatly contributes to my happiness on a day-to-day basis.

And even that much is an overstatement of what the metric measures, because it arbitrarily chooses a cut off for default inclusion into the metric of approximately 30% rejection rate. Why rejection rate of the conference is at all related to which is the best school is completely beyond me, but this means the ranking as actually the order on Number of Papers at Arbitrary Conferences with Below a 30% Rejection Rate Divided by Author Count.

All of these things are proxies, loosely correlated with, but not the same as, quality. Rejection rate is a proxy for quality of a venue, since a highly prestigious venue (not the same as quality, but correlated) will attract more submissions. Since a conference can only accept so many talks, more submissions will be correlated with (but not the same as) a higher rejection rate. Author count is correlated with (but not the same as) a measure of effort per publication, since (in theory) more authors should reduce the effort involved in publishing a paper. Why this particular set of conferences? Well obviously those are “the very top conferences in each area”, according to Emery Berger’s judgement.

So this metric measures SOMETHING, but it’s not what is the best CS grad school, or the most research productive CS department, or really, anything other than “Number of Papers at Arbitrary Conferences with Below a 30% Rejection Rate Divided by Author Count”.

But it is objective and transparent (at least, if you actually spend the time to interpret it and don’t just sort by it). And it does let people make decisions more easily. Not good decisions, but they sure are easy!

We know for sure that Emery Berger’s Ranking has had exactly the effect predicted by behavioural game theory. We have examples of committees using it to judge quality of publication (note: it cannot measure that, see above interpretation). We have examples of people choosing venues not based on the best fit for the work, but to optimize the Emery Berger’s Ranking of their research. Of course they would; that’s the winning strategy when the fitness function uses the Emery Berger’s Ranking of a venue for promotion and tenure (see above).

ICLR Points

Imagine my surprise when a new metric based on the same dataset underlying Emery Berger’s Ranking popped up, and after one look… I LIKED IT.

https://cspubs.org roughly measures the relative effort required to publish 1 paper in terms of ICLR papers. What is this, a metric that seeks to measure effort per publication count? That is actually a thing this data COULD measure. The data takes into account number of authors, which is pretty related to effort. More coauthors mean you can produce more research. It takes into account rejection rate, which is related to effort. More frequent rejections and resubmission require more effort. It takes into account area norms about how much work is required to get a paper at a particular venue, which is pretty much the definition of effort. Since it’s directly measuring effort, it’s not a measure that easily gamed: if you want to increase your ICLR points, then you, what, switch to publishing in venues that require more effort? That will almost certainly net you exactly the same number of ICLR points, since they measure effort.

It’s not a perfect metric. It’s not precise. Is the difference between 1 POPL paper and 1 OOPSLA paper 1.2 ICLR papers? Uh… I dunno, not all OOPSLA papers are the same, not all POPL papers are the same, but maybe? But there is at least a difference in effort.

And yet, I saw people complaining. “Why doesn’t cspubs like ICFP and OOPSLA”. That’s not what it says! It says ICFP and OOPSLA require less effort, which is probably true! They have a lower rejection rate, which decreases effort!! This is quite a good metric for that.

Or maybe I’m just biased because it says PL papers requires the 2nd most effort out of any area in CS. Damn straight I put a lot of effort into my papers.

A high-level summary and interpretation of ACM finances

2023-12-14T20:06:46Z

I’m in the middle of liaising with the UBC library to attempt to negotiate joining ACM Open. It’s not going well. While US universities appear to see cost increases of 2x—3x, Canadian universities are seeing costs of 10x—20x. And our budgets run much tighter, and with far less research funding, particularly for article processing and other publication costs. A 10x—20x increase in publication costs is hard or impossible to swallow.

So, I’ve been forced to look at publishing—where should I publish, how much will it cost, etc—and one question I asked was: why does ACM Open cost so much? It’s a lot more than other open access journals. So I started looking at ACM finances and.. well…

Here’s my dive into ACM finances, and questioning fundamental claims surrounding ACM publishing costs.

UPDATE, March 1 2024 After discussions with other ACM members and feedback on this article, I’ve realized there are several errors in the analysis and the interpretation. I’ll publish an updated version eventually, but leave this version for transparency and as a record.

In short, here are a list of errors in the below facts and analysis:

The data relied upon, FY 2022, is atypical due to including running conferences under COVID restritions. Conferences generally pay for themselves.
The $700 average cost of an article is justified by the Form 990s, despite the profit. This can be seen by cross-referencing the ACM publications finances report with the Form 990s, which is somewhat difficult to do because of the difference in categorizations and granularity of the different reports.
Eliminating membership fees would actually increase costs, since it would likely increase membership, and members receive various benefits which incur costs.
The ACM is probably not as bad as it seems at investing. The calculated return on investment appears to be atypical or artificially low. Further, for very good reasons, the ACM has a conservative investment strategy.
The assets, and income from assets, are at least partially restricted in various ways, both from donors (restricted or endowed funds), or because it is held for the SIGs, which limits how it could be spent down.

However, I still stand by my high-level interpretation of the high profit and high assets of the ACM.

Outdated, invalid information

A high-level summary of ACM finances, from an objective view point

The ACM publishes annual reports in which it summarizes financial information such as its revenue and costs etc. The most recent is available here: https://cacm.acm.org/magazines/2023/4/271230-acm-publications-finances-for–2021/fulltext

Others have relied on this to infer where the $700 APC is spent, and I will argue about that later.

But these reports are presented in the best possible light, to tell a story; they come from a very subjective view point. They’re also at a very low level, so I have a hard time making sense of them at first.

The ACM is also required to file annual IRS disclosures to maintain their non-profit status, which are available with convenient summaries here: https://projects.propublica.org/nonprofits/organizations/131921358.

These are incredibly boring pure figures, presented on PDFs of tables of rows of undigested data—IRS tax documents. Interpreting them can be difficult, for multiple reasons—tax accounting can be different from our intuitive classifications of things; can involve figures from the past and future, anticipated figures, and completely imaginary figures. However, they should be exact and objective.

Here are some salient facts about that ACM’s finances, as reported to the IRS and the public in these filings, and as somewhat inexpertly summarized to a few significant digits by me:

The ACM has been making, on average, ~$8,000,000 in annual “net income” for the past 10 years, and on average $10,000,000 in “net income” for the past 5 years.
The ACM currently has approximately $150,000,000 in investments, essentially all publicly traded securities.
From these investments, the ACM made approximately $3,000,000 in dividends and $3,000,000 in net gains—an approximately 4.4% return on their investment.
The ACM is currently sitting on $25,000,000 in uninvested, non-interest bearing, cash. (I assume it’s filling a swimming poll in Times Square.)
The ACM spent ~$3,300,000 on “good works” related to its mission—grants, student awards, travel awards, etc. This does not include any expenses spent on operating the ACM, publishing, running conference, running the digital library, salaries, etc.
The ACM’s total revenue was $66,000,000, with a breakdown of the major sources below:
- Publishing: $24,500,000
- Conferences: $18,000,000
- Membership fees: $7,500,000
- Ads: $1,000,000
- Investments: $6,500,000
  - Dividends: $3,300,000
  - Capital gains: $3,200,000
The ACM’s total expenses were $56,000,000, with a breakdown of major categories below. There are quite a few other smaller categories, all of which are pretty reasonable.
- Conferences: $20,000,000
- Salaries (non-executies): $7,200,000
- Publication production and services: $6,200,000
- IT: $5,000,000
- “Good works” (grants and awards etc): $3,300,000
- Executives pay: $1,500,000

This summary of the IRS filing is easy to misunderstand. For example, the “conferences” category is probably the most difficult to understand from the IRS filings. Some of this could involve some publication costs, since publications are associated with conferences. The publication costs probably looks artificially low, since the IT category is probably almost entirely related to the digital library, which we would consider part of publication costs, but the IRS considers as a separate line item. Much of the salaries are related to staff for support publication, but salaries are considered separately.

“Good works” is an ill-defined term that comes up in discussion about the ACM’s mission. In the IRS numbers, the things I’ve labeled “good works” are what the IRS calls grants. These are unambiguous expenditures of money by the ACM not on operating any business or service, but that it just gives away. However, note that running conferences, publishing, and maintaining the library are part of the ACM mission.

A subjective interpretation of the high-level ACM finances

Digital library subscriptions and APCs are the main source of revenue

The “publishing” revenue of $24,500,000 is the main source of revenue. We can ignore the conference revenue entirely. This is money that ACM collects ($18,000,000), and then immediately spends on conferences ($20,000,000). The ACM appear to marginally subsidize the cost of conferences, although this is deceptive given the membership fees and the possible misinterpretation of the conference expense category.

Given that they are the main source of revenue, it’s pretty understandable why any change to the structure of that revenue greatly concerns that ACM. A significant change in publication revenue could bankrupt the ACM…. although, it would take several years of literally $0 revenue, given the large pile of reserves the ACM is sitting on.

The ACM is sitting on a very large rainy day fund

The ACM has $25,000,000 in cash and $150,000,000 in investments. Given annual expenses of $50,000,000, the ACM could afford to take in 0 revenue for 3 years, and still wouldn’t be bankrupt.

This seems pretty excessive, but I’m not experienced enough as a financial analyst. The only point of comparison I could quickly find was arXiv, which in 2013 was aiming to cover half of their annual expenses in a rainy day fund: https://info.arxiv.org/about/reports/arXiv_Reserve_Funds_Policy.pdf.

The investments are a little different than the cash, since they could take a while to sell off, you might sell them at a loss, and selling them cuts of a small bit of future revenue. But we’re talking about emergencies and unlikely events here.

The ACM is very profitable, particularly for a non-profit

Normally, a for-profit business would call net income “profit”. The ACM is quite profitable, averaging $10,000,000 per year in profit. It’s not unusual or bad for a non-profit entity to have net income in a given year; it would be a very precarious entity that was constantly breaking even or running a deficit. But this long term trend of surplus is interesting, at least to me, a member, who is being charged various fees that form the revenue of the ACM.

The purpose of a non-profit is not to make profit, but to achieve its mission, which according to Schedule O is:

TO ADVANCING THE ART, SCIENCE, ENGINEERING, AND APPLICATION OF INFORMATION TECHNOLOGY, SERVING BOTH PROFESSIONAL AND PUBLIC INTERESTS BY FOSTERING THE OPEN INTERCHANGE OF INFORMATION AND BY PROMOTING THE HIGHEST PROFESSIONAL AND ETHICAL STANDARDS.

I do not think it is in the public or professional interest to charge unnecessarily high fees, taken out of research and educational funding, and to hold onto that surplus income as an increasingly large pile of assets.

The profit margin for the ACM is 15%. For context, Elsevier has profit margin of around 40%, while Springer has a adjusted operating margin (similar enough to profit margin for our purposes) of 26%. These two are notorious for-profit academic publishers, and should be considered outliers. They have absurdly large profit margins.

For a better view, we could look at various related industries. Given that this is essentially entirely from publishing, I’ll compare this to publishing companies: https://ised-isde.canada.ca/app/ixb/fpd-dpf/report.

From here, I generated the NAICS 513 - Publishing industries - Financial Performance Data, for year 2022, for companies with average revenue $5,000,000—$20,000,000. The publishing industry profit margin range is –8.8%—17.5%.

The ACM has a very high profit margin for the publishing industry.

The ACM is not spending $700 per article

The ACM is not spending $700 per article to publish and maintain the library, no matter how we look at it.

The easiest way to see this is to observe the approximately $10,000,000 in profit the ACM makes, on average, every year, which comes almost entirely from its publication licensing. The ACM could afford to cut the APC to $595, 85% of the APC, giving up its 15% profit margin, and benefiting its members at the same time.

Arguably, it should not give up all net income, as you want some saved for a rainy day. However, the ACM currently has as large rainy day fund.

The ACM funds no good works at all

The ACM funds essentially no good works at all. 5% of the ACM’s costs go to good works, which represents a margin of error in costs. The good works the ACM funds represent the same amount of money, about 3 million dollars, as it makes in annual dividends from its investments. Arguably, nothing the ACM does—collecting revenue from publishing, membership fees, conferences, or donations—funds the good works. It’s only the passive investment in publicly traded companies that fund the good works—the profit from the ACM’s private capital funds good works.

The ACM doesn’t need to charge membership fees

The ACM could cut its membership fees to 0 tomorrow and still be running a $2,500,000 surplus. In fact, probably more, because it could reduce expenses in maintaining infrastructure to collect membership fees, accounting, etc.

The ACM seems very bad at investing

The investment numbers can be very deceptive because its very easy, totally legal, and advisable to play various tax shenanigans with how investments are taxed, so the actual return may be higher.

But, taken at face value, and considering the ~$500,000 in investment management fees, the ACM is extremely bad at investing their money. If they stuck it in a CD right now, they could in principle receive about 5% return for 0 fees. My personal investments currently return about 4.5% dividends and total return of 13%.

If the ACM would like to hire me to manage their money, my rate is $499,000 per year, payable directly to my unrestricted account at UBC. Past performance is no indication of future success; investments may lose value; this does not constitute legal advice; please consult your doctor before starting any medication; etc.

Conclusions: What should I do with this information?

If you’re an ACM member, then there are several things you could call for, press for, attempt to change, etc.

A policy around limiting the net income, or at least directing the use of the net income, of the ACM could substantially reduce costs for members. This could be short-sighted: investing the net income does build an endowment and future income, could better reduce costs in the future. However, the ACM seems bad at investing, and tying the stability of the ACM to the stock market doesn’t seem any more stable than having a long-term balanced budget at lower costs.
A policy to spend down ACM assets. It’s reasonable to keep some cash in reserves, but it may be unreasonable to keep that much cash invested. On the other hand, all universities have large endowments that are similarly managed, so maybe you don’t want to spend it down.
Eliminate membership fees. They represent 75% of the profit.
Eliminate conferences. Conferences are a major expense for the ACM, and apparently a net cost. Removing them could reduce the obvious costs, but also some of the non-obvious indirect costs, and enable reducing publishing costs.
Fund more good works. Maybe we’re fine with the costs, but we just want to spend that profit on good works. We could afford to triple them, and still be profitable.
Do nothing. Maybe things are fine the way they are.

To draw any more conclusions, I think we’d need to dive into the low-level details of ACM’s budget, start comparing it to other publishers, other organizations, reconsidering the services we want the ACM to provide, etc. Some good, low-level information is available in the annual reports: https://dl.acm.org/doi/pdf/10.1145/3389687. One bit that sticks out to me is the cost of copyediting, which is about $1,000,000, and I’d really pay for them to stop destroying my work.

Academia Is a For-Profit Industry

2023-09-18T17:44:45Z

An ill-advised blog post in which I speculate that academia is actually for-profit, but not in the usual sense of “profit”.

Profit is often narrowly construed as about money. It’s not; it’s about value, particularly surplus value. Money is a proxy for value, and often a useful one.

In academia, however, money is not a good proxy for value. I’d argue the primary proxy for value is academia is paper and citation counts, although there are others. And by these measures, academia operates like a for-profit industry with many of the problems that entails: concentration of market power by accumulation of capital, the exploitation of labour, the externalizations of costs, all following the drive for ever higher surplus value. By disguising itself as non-profit, by eschewing money as value, academia disguises these issues.

If paper and citation counts represent value, whence profit? To profit is not to merely produce things of value, but to seek to produce surplus value, and use the capture of that surplus value to incentive production. If papers represent value, then surplus value would represent papers and citation counts beyond what is required to produce the embodied knowledge. If academia were for-profit, we’d expect structural incentives to produce more and more papers and citations, and to produce them more efficiently (or at lower cost anyway). We’d expect to see that academic institutions measure success by increasing profit (i.e., papers and citations). That, for example, academics are hired and retained based on their ability to bring profit to the employing institution. We would expect capital to accumulate that makes increasing profit easier, so some groups accumulate profit that they can reinvest into increasing numbers of papers and publications.

Some of these are easy to test; the others are testable with some straightforward data collection and analysis. I’m not going to do that now because (1) I’m bad at numbers (2) research is my job and no one is paying me for this.

Academics and universities do judge their success, at least in part, by the number of papers and citations. See for example https://csrankings.org for one formalization of this; it’s like the Dow of CS universities. Or something. Researchers are hired and rewarded based on publication count and citations. These go into the hiring packets for new faculty and faculty up for tenure. Some contracts have specific publications counts you have to hit to make tenure. This is analogous to high-level management positions with profit targets; missing them could well get you fired.

On it’s face, this isn’t necessarily a problem, and in some ways, is less problematic than a for-profit economy in general. At least papers represent knowledge, so we’re incentivizing the production of knowledge. The reasons I dwell are: (1) an incentive to optimize a proxy, a metric, is IMO bad; it leads to perverse incentive, i.e., incentives to optimize merely the metric at the expense of the thing measured. This might not be a problem but for: (2) while papers represent knowledge-as-value, this value can be exchanged for monetary gain, so exploitation of the paper metric is profitable in the usual sense, and at the expense of knowledge.

So, does the “profit” motive lead to exploitation? I’d say so, and in the same predictable ways as the usual profit motive. The easiest ways to increase production and decrease costs aren’t do hard work at making things more efficient, but to lie, cheat, and steal. We see these are work in academia. Made up data, plagiarized papers, etc. But let me zoom in on some higher level stuff.

Exploitation of labour happens, at least until tenure. Grad students are often exploited and receive only a part of the value of their labour, although interesting I’d argue that by the measure of paper and citation count, less exploited than other industries. Grad students usually receive an equal share of the paper count and the citations. However, they might put in a disproportionate share of the labour required to produce a paper. I’ve heard of cases where students are cut out of a paper they worked on, or a supervisor puts almost no work in but receives a paper and citations. Very capitalist—they provided the capital to produce the paper, even if they provide no labour, and receive a share of profit in return. Importantly, though, junior faculty (at least) are exploited as well. Here, it’s not by a direct superior, but indirectly by the institution, through a series of committees of “peers”. “Publish or perish” means they must produce value, papers, or be replaced by the many workers willing to step into the scarce high-demand faculty positions. Even highly respected workers who miss arbitrary targets are denied tenure, because they don’t produce enough profit. Exploitation of the individual worker only ceases if they get tenure, although systemic exploitation continues. In this view, I guess tenure is finally accumulating enough capital to become a capitalist, and live off the rents of capital accumulated? You never have to produce another paper again. Here, the exchange of “profit” for profit seems important.

The analogy to capital accumulation holds as well. The more publications and citations you get, the easier it is to get more. To produce more and more citable papers requires grant funding, top graduate students, and post-docs. To get these, you need to have sufficient grant funding, and sufficient reputation and renown to attract top workers. In a typical for-profit industry, those with the most profit can afford to recruit the best talent, by paying more. In academia, though, salaries as in money are relatively fixed by institutions, not individual advisors (exceptions exist). Instead, academics compete largely on reputations—who can train you to produce, and get you the most, high quality publications. That is, who can help you acquire the most capital-of-academia. Profit should also be translatable into capital, and this happens. Grants and awards look, in part, at publications and citations (and, ironically, other awards received) to decide who gets awards. These grants and award fund more papers and citations. Higher profits lead to the accumulation of capital, which leads to higher profits. Metrics like the H-index even ensure papers and citations depreciate, unless they generate ever higher citations over time.

Costs of production get externalized. The most notable might be, as in many industries, greenhouse gasses. To produce papers and make them well known to gain citations often requires travelling, often internationally, often frequently. One other I worry about, although it’s unclear whether it’s a real problem, is the training of large numbers (I’d argue, unnecessarily large numbers) of PhDs. I’d argue we produce more than academia needs, except that we need them to produce research cheaply. In some places, there is pressure to increase the size of the PhD program. There’s a straightforward way to reduce these costs, and like in other industries, there’s no way in hell we’re doing it: produce less, or produce slower, or both. Of course, like in other industries, there’s reason. If we do stop producing sufficient profit (papers), people will lose their livelihood. Grad students may not get jobs if they don’t produce enough papers presented at international venues. Junior faculty may not get tenure.

I don’t think any of this is a particularly novel observation, except the explicit analogy as profit. And even that I doubt.

And the analogy isn’t perfect, and it’s complicated by the intersection with the money economy. That’s okay; no model is perfect, but some are useful. I think this for-profit model of academia is useful.

What is a model?

2023-06-15T20:25:25Z

What is a model, particularly of a programming language? I’ve been struggling with this question a bit for some time. The word “model” is used a lot in my research area, and although I have successfully (by some metrics) read papers whose topic is models, used other peoples’ research on models, built models, and trained others to do all of this, I don’t really understand what a model is.

Before I get into a philosophical digression on what it even means to understand something, let’s ignore all that and try to discover what a model is from first principles.

Definitions of “model”

The apparent place to start to understand the meaning of a word is to read its definition. This is actually no help at all. There are lots of uses of the word “model”, with several definitions. Here are some.

Definition 0 In science and engineering, a model is “an abstract description of a concrete system using mathematical concepts and language”. See Wikipedia provides a nice introduction to this kind of model, and the Standard Encylopedia of Philosophy provides a nice explanation in the context of model theory, which will be relevant later in this post.

Definition 1 A syntactic model (of a type theory) is defined by Boulier, Pédrot, and Tabareau as a translation from one type theory into another that preserves typing, the definition of false, and definitional equivalence. This syntactic model enables the source type theory to inherit properties of the target type theory—such as consistency.

Definition 2 A model (of a vocabulary also called a language $\sigma$) in the sense of model theory (as defined by Elements of Finite Model Theory) is a $\sigma$-structure (“also called a model”) defining a set A along with 3 sets providing interpretations of that vocabulary. These sets are $Ic_A$, which interprets each constant in $\sigma$ as an element of $A$, $IP_A$, which interprets each n-ary predicate symbol or relation symbol from $\sigma$ as an n-ary (set-theoretic) relation between elements of $A$, and $If_A$, which interprets each n-ary function symbol in $\sigma$ as a (set-theoretic) function from n elements of $A$ to an element of $A$.

Definition 3 The above definition is confusing, since it conflates structure and model, which the text later distinguishes with the following separate definition. A model (of a theory (over a vocabulary $\sigma$)) is a structure (“also called a model”) of vocabulary $\sigma$ such that every sentence in the theory is interpreted in the structure to make the sentence true. (A theory is a set of sentences drawn from a vocabulary.) My rephrasing of the definition of model is intentionally confusing and difficult to parse, to make apparent the inherit confusingness created by the several layers of definitions and one definition that defines “model” using a second definition of “model”.

Definition 4 Nlab hosts an article with a much clarified definition, which distinguishes language, theory, structure, and model carefully. In particular, it is careful to only call structure the interpretation of the language (call vocabulary above), and only call model an interpretation that makes true the axioms composing the theory of the language.

Definition 5 Carlo Anguli once gave me the following definition of model:

A collection of interpretation functions that interpret every syntactic category in such that the original relationship is respected.

e.g.,
- interpret every context as a set,
- interpret every (non-dependent) type as a set, and
- interpret every term-of-a-type indexed-by-a-context as an element-of-the-interpretation-of-that-type indexed-by-elements-of-the-interpretation-of-that-context.

Implicit in this definition is that the interpretations must respect equality — because if you don’t respect equality of arguments then you’re not a function!

This definition seems to be close to Definition 2, as it doesn’t mention axioms and their interpretation. However, it might be Definition 3 instead, as there could be implicit in the definition of syntax an inclusion of all judgements of the programming language, and therefore in the phrase “such that the original relationship is respected” a requirement that axioms of those judgements become true.

Also implicit in this definition of model is what it is a model of. Perhaps a programming language, but it again depends on what syntax means and thus what “every syntactic category” refers to.

I’m interested in the requirement that the interpretation is a collection of functions, which seems to be missing or only implied in some model theory definitions of “model”.

So what is a model?

One of the first thing that jumps out to me after reviewing the above definitions is that to understand each definition, you have to reframe the definition of model into model [of what]. It really never makes sense to give a definition of merely “model”.

Definition 0 defines a model of a system [of the real world]. Definition 1 defines a model of a type theory. Definitions 2 and 3 give definitions of a model, in the sense of model theory, but of two different objects: model of a vocabulary (or language), which is more often called a structure, and model of a theory (which everyone seems to agree is a “model”). Definition 4 makes this distinction very clear. Definition 5 seems to use “model” in the model-theoretic sense, but has abstracted a bit away from a particular notion of theory and generalized to syntax.

What is a model of a programming language?

I’ve had two problems understanding the word “model” in the context of programming languages.

First, we use “model” in three different senses, and I have neither understood that nor understood the relationships between them.

Model in the sense of an abstract description of a system. This is Definition 0. This sense of “model” means something like “mathematical description”. What we want is a description in which we can work using math, so we can make predictions about the real world. Ideally, the predictions we make will be true.
Model in the strict sense of model theory. These are Definitions 2, 3, and 4. This sense of “model” is the closet to having a strict definition. It often carries a set-theoretic connotation, asking for a set defining the domain of values, and three interpretation functions that interpret specifics parts of a theory in specific ways.
Model in the generalized sense, inheriting from or related to model theory. I hesitate to even call this distinct from the second sense, but I will anyway. I also hesitate to speculate about history—perhaps this sense actually predates model theory. But I distinguish it from the second sense, because it frequently generalizes away from the strict 3 category “constant symbol”, “predicate symbol”, “function symbol” specification and doesn’t seem beholden to set theory. Definitions 1 and 5 use “model” in this sense.

In the second sense of “model”, the first sense of the word remains—we’re still interested in a description of some system (the theory), and of using the model to make predictions or reason. However, since the theory is also mathematical, we can be more rigid about our reasoning requirements—axioms of the theory must be true of the model, and relationships must be preserved in the model. This is rarely true of a model of the real world; e.g., the Newtonian model of gravity works pretty well, until it doesn’t, so it’s a model that doesn’t quite make all axioms true or preserve all relationships.

The third sense seems closer to the idea of semantics, in the mathematical logic sense of the word as assigning meaning or interpretation to syntax. In this sense, the word “model” frequently avoid committing to set theory as a formal foundation, generalizes away from the three interpretation functions, and focuses instead on the relationships between uninterpreted syntax being preserved by the interpretation. For example, in Definition 1, the relationships of interest are well-typedness, definitional equivalence, and falsehood, and the formal foundation is type theory. Category theory seems to come closest to a complete formalization of this sense of the word “model”, although I’ve had a hell of a time understanding that. Nlab articles don’t say this explicitly, but reading between the lines in articles linked to from the Nlab article on model theory for the words syntax and semantics implies that the idea of syntax, i.e., uninterpreted symbols with relationships between themselves and judgements about them, can be formalized in category theory, and then so can the idea of semantics, i.e., providing an interpretation in some other domain of those uninterpreted symbols; a domain in which one can use all the power of the other domain to reason about the judgements one wishes to make about the uninterpreted symbols.

The second problem with the word “model” is that we frequently work with two senses simultaneously.

When I write down a programming language, I’m often trying to model (in the first sense) a real programming language (or some feature of it), one actual software developers use to make real things happen in the real world. I am not merely describing a mathematical object for study. (Okay, sometimes I do that, but usually to the first end, eventually.) When I write down such a model, I may describe the abstract syntax, the typing judgement, and an abstract machines or reduction rules. These form a pretty good mathematical description of how a real language behaves. A compiler will reject syntactically invalid expressions. It may then type check the abstract syntax tree, and reject some possibly semantically invalid expressions. If judged well typed, the compiler may transform the tree into something that runs, and that run-time behaviour can be predicted using the reduction rules.

However, for much programming languages work, I’m not interested in merely predicting the behaviour of a single program. I might want to predict behaviour or properties of the entire language, or its typing judgement, etc. To reason about single programs, the model (in the first sense) may work well. But it might not work well for, say, trying to decide whether certain types can even be inhabited. To solve this, we might build a model (in the second or third sense). We interpret the abstract syntax tree and typing judgement in some other domain. That is, the AST and the typing judgement, being a model in the first sense, form a theory in the model theoretic sense. We can then construct a model (in the second sense) of a model (in the first sense). The Standard Encyclopedia of Philosophy article on model theory goes into this in detail in the context of model theory, which is great.

What’s more interesting is how these two senses of model interact in programming languages. If one is interested in a model, in the second sense, it may inform how one develops a model (in the first sense). If I know I will want to construct a model (in the second sense) to reason about the typing judgement, I may decide that single-step reduction rules are actually irrelevant; I only care that certain program equivalences hold, really, and any implementation that has those equivalences suffices. So rather than create a model (in the first sense) with an abstract machine or small-step operational semantics, I’ll specify an equivalence judgement. This might give less predictive power about a real world implementation, but allow the predictions I do make to apply to many implementations.

If you see these patterns, you may have some insight into how the author is approaching their work, and in what senses they are using the word “model”.

What is syntax?

2023-06-07T20:58:46Z

I’m in the middle of confronting my lack of knowledge about denotational semantics. One of the things that has confused me for so long about denotational semantics, which I didn’t even realize was confusing me, was the use of the word “syntax” (and, consequently, “semantics”).

For context, the contents of this note will be obvious to perhaps half of programming languages (PL) researchers. Perhaps half enter PL through math. That is not how I entered PL. I entered PL through software engineering. I was very interested in building beautiful software and systems; I still am. Until recently, I ran my own cloud infrastructure—mail, calendars, reminders, contacts, file syncing, remote git syncing. I still run some of it. I run secondary spam filtering over university email for people in my department, because out department’s email system is garbage. I am way better at building systems and writing software than math, but I’m interested in PL and logic and math nonetheless. Unfortunately, I lack lot of background and constantly struggle with a huge part, perhaps half, of PL research. The most advanced math course I took was Calculus 1. (well, I took a graduate recursion theory course too, but I think I passed that course because it was a grad course, not because I did well.)

So when I hear “syntax”, I think “oh sure. I know what that is. It’s the grammar of a programming language. The string, or more often the tree structure, used to represent the program text.”. And that led me to misunderstand half of programming languages research.

The First Meaning of Syntax

Syntax has two meanings in programming languages, and both meanings can frequently be found in the same paper.

The first meaning is the one I gave above. I could give a definition of the syntax (in the first sense) of the lambda-calculus as follows.

e ::= x | (lambda (x) e) | (e e)

Ah. Beautiful syntax.

If we were following a standard text, such as Harper’s Practical Foundation for Programming Languages (2nd ed), we might next define the “semantics” of this “syntax”. We might define the “static semantics”, i.e., the type system or binding rules, then the “dynamic semantics”, i.e., the rules governing the evaluation behaviour of the syntax. For example, I might write the following small-step operational semantics.

((lambda (x) e) e') -> e[x := e']

Ah. Beautiful semantics.

Except, everything I wrote above, reduction rule included, is also syntax and not semantics.

Historical Interlude

The words “syntax” and “semantics” come from mathematical logic.

In that context, “syntax” describes sentences, statements, symbols, formulas, etc, without respect to any meaning. You can write down a logical formula say as "∀ X.P(X, A)" (where “A” is a logical constant, “X” is a variable, “P” is a proposition), and it has no meaning; it’s mere syntax. It might be true, or might be false, depending on its interpretation of “P”, “A”, and "∀". I could say that it means “all leaves are green”, which would be false. A more relevant example for PL might be the syntax ((lambda (x) x+1) 2) = 3, which I would certainly like to be true, but it very much depends on what I mean. If + means string append as in JavaScript, then the statement is false since ''.concat(1, 2) = '12'. Wikipedia is a good start for trying to understand this history of the word “syntax”: https://en.wikipedia.org/wiki/Syntax_(logic)

By contrast, in that same context, “semantics” is the means by which syntax is given an interpretation. Perhaps the most widely used approach to providing an interpretation of syntax is model theory, which I never learned. In model theory, we start with a “syntax” (or “theory”). This theory is a collection of constants, function symbols, and predicate symbols. A model then is a map from the uninterpreted syntax to some interpretation that preserves relationships. I’ll say more of this in a later post, but for now, consider the following example. I might provide a model of our earlier example that interprets + as ''.concat, and = is mapped to, say ===. This preserves relationships, if all my constants are mapped to strings. Wikipedia is a good source for this history too: https://en.wikipedia.org/wiki/Semantics_of_logic.

When Semantics is the Syntax

What’s interesting about this history is how it was adopted in programming languages, and evolved in two different ways. On the one hand, a programming language grammar is syntax, in the sense of being uninterpreted statements. That syntax can be given a semantics, an interpretation, by using operation semantics (this is the sense in which operational semantics is a semantics). The operational semantics provides an interpretation to our grammar.

But, in another sense, the grammar, typing rules, and evaluation rules (the “syntax”, “static semantics”, and “dynamic semantics”) are mere syntax, in the older logical sense. They are a theory, in the model-theoretic sense. To see why, we must understand what the earlier example ((lambda (x) x+1) 2) = 3 means. Or in fact, realize that it doesn’t mean anything at all.

To write this down is to write down a proposition about the grammar: that one piece of the grammar is equal to another. Except I didn’t write a proposition that the two were equal. I wrote the uninterpreted proposition symbol =, the syntax =, next to two pieces of uninterpreted grammar, two other pieces of syntax. Every syntactic judgment about our grammar is itself syntax, in the model theoretic sense. At least, this is true if we follow the tradition of writing them down synthetically, axiomatically, about the grammar, as is done in standard programming languages textbooks such as Types and Programming Languages or Practical Foundations for Programming Languages.

In this view, the typing rules and reduction relations are syntax. This is a bizarre perspective from a software engineering perspective, but makes sense from the mathematical logic perspective.

With this perspective, it might make sense to call “operational semantics” “syntactic semantics”, or to imagine a tower of syntax and semantics where one level’s semantics become the next level’s syntax. This view finally helped me make sense of why we call “syntactic logical relations” syntactic, when they are clearly semantics. (A problem I danced around in my previous post on logical relations.)

This perspective is also useful, for two reasons. The first is that reasoning purely syntactically, while very general, prevents you from importing any other reasoning principles from any other domain. By viewing the typing system as syntax, and then building a model of it (and by necessity, the programming language terms) in, say, set theory, we can import all set-theoretic reasoning in our attempts to reason about our type system. But more than that, we can reinterpret the syntax freely, to prove general results. While I might have written a type system using syntax that looks like numbers, I could build a model that interprets that type system as over strings, and know that actually the entire system is safe for strings, too. Appropriately generalized, I wouldn’t need to do any additional proofs.

Unfortunately, this double meaning of the word syntax seems to be completely taken for granted by some. nLab is a good example of this. To quote from the introduction to the nLab model theory page:

On the one hand, there is syntax. On the other hand, there is semantics. Model theory is (roughly) about the relations between the two: model theory studies classes of models of theories, hence classes of “mathematical structures”.

What’s most interesting about this quote isn’t what it says, but what it links to. The link for “syntax” is to the page on the internal logic of a category. From the software perspective, this is not syntax, but semantics. How on earth could it be syntax? The link for “semantics” is to the page on structure, the idea of equipping a category with a particular functor. How on earth is that any more semantics than the original abstract nonsense version of syntax?

Before I understood “syntax”, I couldn’t make any sense of that, but now I’m beginning to understand. The internal logic of a category in some sense must be able to express the grammar of a language, and the judgments of a language, but in a purely syntactic way—in the same way that when I write down the grammar and typing rules of a language, I don’t refer to any interpretation of those symbols beyond the way I combine them on the page. Then the semantics or structure is a the particular functor over that category, providing an interpretation, a semantics, of that original category (the syntax).

Anyway, now I think I’m ready to understand what a model is.

In What Sense is WebAssembly Memory Safe?

2023-05-19T02:35:56Z

I’ve been trying to understand the semantics of memory in WebAssembly, and realized the “memory safety” doesn’t mean what I expect in WebAssembly.

What is memory safety?

Here are some definitions.

Memory safety is a feature of programming languages that prevents certain types of memory-access bugs, such as out-of-bounds reads and writes, and use-after-free bugs. In an app that manages a list of to-do items, for example, an out-of-bounds read could involve accessing the nonexistent sixth item in a list of five, while a use-after-free bug could involve accessing one of the items on an already deleted to-do list.

https://spectrum.ieee.org/memory-safe-programming-languages

Memory safety is the state of being protected from various software bugs and security vulnerabilities when dealing with memory access, such as buffer overflows and dangling pointers. For example, Java is said to be memory-safe because its runtime error detection checks array bounds and pointer dereferences.

https://en.wikipedia.org/wiki/Memory_safety

Memory (un)safety in Wasm

WebAssembly (Wasm) is a language that guarantees “type safety … [preventing] invalid calls or illegal accesses to locals, … memory safety, and … inaccessibility of code addresses or the call stack”.

(Technically, the Wasm paper describes Wasm as a binary code format, that happens to be presented as a language.)

Formally, a whole Wasm program that type checks is guaranteed to either be a well-typed value, or take an evaluation step to a well-typed program, or evaluate to the well-known dynamic error “trap”.

This is in contrast to an unsafe language like C. A well-typed C program might take a step to a well-typed program, or it might evaluate to a value of arbitrary type or no type. For example, a well-typed program of type char that reads from a buffer might evaluate to a well-typed char, or it might evaluate to an arbitrary integer that does not correspond to any character because you were reading uninitialized memory.

For example, consider the following C program.

// unsafe.c
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char** argv) {
  char* buf = malloc(0);
  memcpy(buf, "Hello world\n", 12);
  write(1, buf, 12);
  return 0;
}

(compiled with clang -o unsafe.exe unsafe.c; run with ./unsafe.exe)

This program creates a buffer of size 0, writes “Hello world\n” to it, and tries to print that to standard out. The program printed “Hello world” when I ran it, but it’s undefined behaviour, so anything could happen. I tried to writing a loop that mallocd lots of memory and wrote arbitrary numbers, but never managed to crash the program. Still, it’s not memory safe.

The equivalent Wasm program is below.

;; safe.wat
(module
 (import "wasi_unstable" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32)))

 (memory 0)
 ;;(memory 1)
 (export "memory" (memory 0))

 (data (i32.const 0) "Hello World\n")

 (func $main (export "_start")
       (i32.store (i32.const 12) (i32.const 0))
       (i32.store (i32.const 16) (i32.const 12))
       (call $fd_write (i32.const 1) (i32.const 12) (i32.const 1) (i32.const 20))
       drop))

(run with wasmtime safe.wat)

In this example, we create a string “Hello World\n” at address 0 in the module’s memory. We then create (encode) a new iovs just after it, starting as address 12, with a pointer to address 0 and length 12. Then we call fd_write, from the wasi API.

Unfortunately, we declared the memory size to be 0, so trying to allocate this string fails, traps safely, and the process exits with an error message.

So wasm is memory safe right?

Well, sort of, but there’s a pretty key distinction here.

In C, we are creating a new pointer with malloc. We are allocating a new data structure, then using it (unsafely).

In Wasm, there is exactly one memory for the entire module. Inside that memory, we encode 2 data structures: our string, and the iovs structure used by fd_write. All access to the global memory are safe. But not all accesses to the encoded data structures are.

Most application will create data structure within the memory. That’s what our call to fd_write did. The two stores actually create an iovs structure in the global memory. We have no guarantees, within Wasm, about that data structure.

For example, here’s our Hello World program in Wasm which uses the memory safely and correctly, but creates an iovs whose length is claimed to be 100, larger than the actual string.

;; unsafe.wat
(module
 (import "wasi_unstable" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32)))

 (memory 1)
 (export "memory" (memory 0))

 (data (i32.const 0) "Hello World\n")

 (func $main (export "_start")
       (i32.store (i32.const 12) (i32.const 0))
       (i32.store (i32.const 16) (i32.const 100))
       (call $fd_write (i32.const 1) (i32.const 12) (i32.const 1) (i32.const 20))
       drop))

(run with wastime unsafe.wat)

When I run this, I get “Hello world\nd” printed to stdout. I have no idea where that trailing d comes from, and it didn’t crash, suggesting it read uninitialized memory of some kind.

Arguable, this is cheating: Wasm does not and cannot make claims about external system functions, and wasi is unstable. But IMO the root of the error isn’t really about wasi.

Really, the root cause of this error is memory unsafety, but of a data structure encoded within a Wasm module. In a truly memory-safe language, if I try to access the 100th element of a 12-character long string, I get an error:

> racket
Welcome to Racket v8.9 [cs].
> (string-ref "Hello world\n" 100)
; string-ref: index is out of range
;   index: 100
;   valid range: [0, 11]
;   string: "Hello world\n"
; [,bt for context]

But that doesn’t happen in Wasm.

Wasm memory safety doesn’t apply to data structures implemented (encoded) within the memory. It only applies to the module’s memory, which is protected from other modules, even those running in the same process’s virtual address space.

This means Wasm modules are protected from each other, and so this kind of memory unsafety probably isn’t a security risk, only a cause of logic bugs.

In Wasm, data structures have to be encoded anyway, since Wasm doesn’t provide any kind of structure data primitives; you only have integers and some integers are interpreted as addresses into memory. But, when you encode such data structures in the memory and use them incorrectly, you have no guarantees about what happens. You could read some arbitrary data (from your own module), or read some uninitialized memory (from your own module). I.e., you get out-of-bounds reads and writes.

In another view of this, memory is the only data structure in Wasm, and it is memory safe. That’s all the language can be responsible for; if you go about encoding weird things inside that data structure, errors are likely. But this doesn’t seem like what people would expect when they hear “memory safe”. At least, it’s not what I expected at first.

What is logical relations?

2023-03-24T23:32:03Z

I have long struggled to understand what a logical relation is. This may come as a surprise, since I have used logical relations a bunch in my research, apparently successfully. I am not afraid to admit that despite that success, I didn’t really know what I was doing—I’m just good at pattern recognition and replication. I’m basically a machine learning algorithm.

So I finally decided to dive deep and figure it out: what is a logical relation?

As with my previous note on realizability, this is a copy of my personal notebook on the subject, which is NOT AUTHORITATIVE, but maybe it will help you.

Here’s my working definition of a logical relation:

A realizability semantic model,
built of predicates over syntax,
that reflects judgments and structures from semantics to syntax.

Point 1 is subtle; it implies that the logical relation is both a model, and a realizability semantics. Unfortunately, I still don’t know what a model is, so I’m going to have work with the following probably wrong oversimplification: the logical relation must take (syntactically) equal terms to (semantically) equal terms. Which notion of syntactic equality though? I’m not sure, and I’m going to ignore it for now.

Point 2 is actually more specific than necessary. We don’t need to predicates over syntax specifically, but really over some base model. It’s easier for me to think of this as “syntax”, though.

Point 3 is quite difficult to make precise without making a lot of this more precise in a mathematical framework. Jon Sterling gave me the following helpful definition:

A logical relation on a model M (viewed as a category) is then a model that is constructed in the following way:

Choose some functor R : M —> E where E is a sufficiently structured category (e.g. the category of sets, or something else!). The most basic example of a functor R is the “global sections functor” M —> Set, which sends every type in M to the set of closed elements of that type. This is exactly the usual “non-Kripke logical relations"; to get Kripke logical relations, you replace Set with a functor category (presheaf category) and choose a more interesting functor R.

Now define a new category G, as a category whose objects are pairs of an object A of M, together with a subobject of R(A). A morphism in G from (A,A’) to (B, B’) is given by a morphism (f : A -> B) that sends elements satisfying A’ to elements satisfying B’.

You have to show that the category G is actually a model of your language (e.g. show that it has function spaces, booleans, whatever). Doing so is the FTLR.

Note that there are some ways to generalize the situation above, but this is basically what logical relations are.

Point 3 is also is more specific than necessary; “syntax” can be generalized to be “base model”.

Despite the complexity, we can see point 3 in action in some examples below.

What are logical relations, historically?

tait1967 - Intensional interpretations of functionals of finite type I

Logical relations are sometimes called “Tait’s Method”, dating back to Tait, as far as I can tell.

In this paper, Tait proves that System T with bar induction is a conservative extension of intuitionistic analysis U_1, which is intuitionistic arithmetic plus quantification over functions plus the axiom (schema) of choice plus bar induction. This conservative extension property is the semantic property of interest. The proof starts with a proof that T without bar induction is a conservative extension of just intuitionistic arithmetic (no choice or bar induction).

To do this, Tait develops a type-indexed predicate over System T terms (without bar induction), providing a U_0 term for all T terms of each type. These predicates M_t, C_t, and E_t are (I think) what we refer to as a logical relation. In particular, the C_t relation provides the interpretation of T values of type t, M_t seems to deal with variables, and E_t seems to be a binary relation defining semantics (“weak α-definitional equality”) of terms.

Theorem V (page 205) uses this logical relation to prove that, for all semantics values at the same type, (weak α-) definitional equality is decidable: they either are or are not related in E_t. This seems to be the key point: the definitional equality is reflected out of the semantics of terms, so it can apply to the syntax of terms.

This use of logical relation seems to also be a realizability semantics, since it it assigns syntactic types to a collection of semantics terms, by induction over syntactic types, where the realizers are a subset of all possible semantics terms.

However, it seems to be more than a realizability semantics, too. What seems very important in this paper is that the semantics preserves structure, namely definitional equality. Perhaps implicitly though, other pieces are important. For example, T functions are interpreted as U functions, although it’s not clear to me that this is critical.

This is in contrast to Kleene’s (kleene1945) realizability, which did not seem concerned with structure, but only the existence of the realizers.

plotkin1973 - Lambda-definability and logical relations

Plotkin seems to be responsible for the name, and perhaps rediscovering logical relations in the context of programming languages.

Plotkin helpfully gives us a definition of “logical”, as well, and it seems quite importantly related to part 3 of my working definition. Plotkin defines a relation R as logical if it is:

a subset of any D_k from the carrier any D∞ model (this seems to correspond to “admissible relations” in modern logical relations parlance);
the relation is preserved by functions in D. That is, the relation holds on a function f in D_k iff for all arguments x, R(x) implies R (f x) (extended to the n-ary case for n-ary relations).

This suggests that it is important the logical relation is somehow interpreting syntactic structures as semantic structures, as in the case of Tait’s model interpreting syntactic functions as semantic functions. More generally, we likely want this property of all structures in the languages: syntactic pairs are interpreted as semantic pairs, etc. Jon’s category theoretical definition seems to generalize Plotkin’s definition nicely.

This denotational logical relation also shows us a logical relation that is not defined over syntax. Instead, it is a relation over some arbitrary non-trivial D∞ model. The author mentions that since they can interpret syntax in a D∞ model, they informally treat the logical relation as over syntax sometimes, which I suppose could be made formal easily enough.

How is “logical relations” used in PL?

ahmed2006 - Step-Indexed Syntactic Logical Relations for Recursive and Quantified Types

In this paper, Ahmed is concerned with syntactic logical relations for recursive and quantified types, in particular for reasoning about contextual equivalence. Likely due to Ahmed’s work, this kind of syntactic logical relation seems to be what most people mean or think when they say “logical relation”, although that may be changing.

The desired property of the logical relation then is that two related semantic terms should be contextually equivalent in the syntax. That is, the logical relation reflects (from semantics to syntax) equivalence.

Strangely (for a realizability model), this particular syntactic logical relation also reflects typing: semantic terms in the relation are also guaranteed to be well-typed in the syntax. In contrast, some uses of “logical relations” enable semantics terms to be syntactically ill-typed. Such logical relations might be better called realizability models, although they do something reflect some structure, so perhaps reflecting typability is not a critical point of reflecting structure.

Ahmed in the introduction points out an interesting distinction: that logical relations can be either denotational, or syntactic. Syntactic logical relations model syntax as sets of syntactic values such that some property holds over that syntax. By contrast, denotational logical relations model syntax as some denotational object, Syntactic logical relations are useful for proving properties of the operational semantics directly. Denotational models instead model syntax as denotational objects, such as, e.g., sets of set-theoretic functions over elements of a D∞ model in plotkin1973. This is useful for easily proving meta-theoretical properties by reflecting properties of the denotation into the syntax, but not necessarily about the operational semantics directly.

For example, Tait uses a “denotational logical relation” into intuitionistic analysis to prove that definitional equality of System T is decidable—the definition of definitional equality, in the model, and its proof of decidability, are reflected back into the syntax; this requires no operational semantics at all. Plotkin uses a denotational logical relation, into domain theory, to show that certain λ-calculus constructs are or are not definable—existence of a term in the logical relation is reflected into the syntax as a definable expression. Neither of these is a syntactic logical relation; the semantic values never mention syntactic values directly.

Ahmed uses a “syntactic logical relation” to prove something about the operational semantics, namely, to prove contextual equivalence (an operational notion), indirectly. Direct proofs of contextual equivalence are difficult. So instead, a semantic proof of equivalence is reflected back into the syntax as ta proof of contextual equivalence. This requires structuring the logical relation into a denotation of sets of syntactic terms that evaluate in the operational semantics, so that being in the relation tells us something about evaluation in the operational semantics, which tells us something about contextual equivalence.

abel2018 - Decidability of conversion for type theory in type theory

Abel et al. define a syntactic logical relation for typed, reducible (and equivalent) terms, to prove decidability of conversion for type theory. Here, the use of syntactic logical relation is important for proving a particular conversion algorithm over the syntax is decidable.

The interesting feature of this logical relation is the generalization from a model inductively defined over types, to inductively defined over judgments. This demonstrates a weakness in my working definition of logical relation and realizability, since I defined “realizability” in terms of models inductively defined over types.

timany2022 - A Logical Approach to Type Soundness

This paper is interesting because it uses a syntactic logical relation that intentionally does not reflect typing, as many syntactic logical relations cdo. Semantically valid terms are not necessarily syntactically valid. In other ways, it looks very much like a logical relation: syntactic pairs are semantic pairs, sums sums, functions functions, etc.

The key property this paper is interested in is type safety: all well-typed terms are well-defined in the operational semantics, i.e., they evaluate to values or well-defined errors or fail to terminate, but importantly, do not get stuck. “in the operational semantics” is important to understanding why this is a syntactic logical relation; it must model terms as sets of syntactic values to reason about the operational semantics given in the paper.

However, one could imagine proving a slightly different form of type safety with a denotational logical relation. Giving a logical relation into an arbitrary model with a well-defined notion of evaluation would be implicitly a proof of type safety: that there exists a model that is type safe. The ability to reflect from semantics to syntax provides a mechanism for constructing that evaluation over syntax. So while the denotational logical relation provides no direct proof about the operational semantics, it may provide a mechanism for a type-safe-by-construction operational semantics. (This reflecting evaluation out of the semantics seems very related to the idea of normalization-by-evaluation, but I’m not clear on this.)

What is realizability?

2022-10-05T21:54:39Z

I recently decided to confront the fact that I didn’t know what “realizability” meant. I see it in programming languages papers from time to time, and could see little rhyme or reason to how it was used. Any time I tried to look it up, I got some nonsense about constructive mathematics and Heyting arithmetic, which I also knew nothing about, and gave up.

This blog post is basically a copy of my personal notebook on the subject, which is NOT AUTHORITATIVE, but maybe it will help you.

My best understanding of realizability right now, in programming languages (PL) terms, is:

A technique for assigning each syntactic type to a collection of semantic terms;
By induction over syntactic types;
Where the semantic terms that are realizers—i.e., included in the collection related to some syntactic type—are a sub-collection of all possible terms in the semantic domain. That is, there are valid semantic terms not associated with any syntactic type.

I use the word “collection” rather than “set” to avoid invoking set theory.

Graphically, we can represent this as follows:

The point of the technique is that clause 2 gives us a proof technique by induction, and clause 3 means we can relate the collection of terms (or proofs) to some other well-known collection. This yields a proof technique for metatheoretic properties about the collection, such as that there are only terminating terms in the collection of realizers, or there are only recursive functions and therefore some classical things remain unprovable.

I’m not entirely sure that clause 2, induction, is necessary, and I can’t find anything explicit about clause 3, but they seem to be true historically and in many uses of the term.

Okay so how did I get to this understanding?

What is realizability, historically?

kleene1945 - On the Interpretation of Intuitionistic Number Theory

Realizability seems to come from Kleene’s paper “On the Interpretation of Intuitionistic Number Theory”. I say “seems to” as Kleene attributes the “detailed investigation of the notion of realizability” to David Nelson, attributes several of the results in the paper to Nelson, and claims that the main results of the paper are joint work with Nelson. But the paper only has Kleene’s name on it, and Kleene claims in the first footnote that they introduced the idea of realizability to Nelson in a seminar. So anyway, realizability seems to come from Kleene, and this is the canonical paper cited for the technique.

In this paper, realizability is quite specific. It’s a technique that takes an intuitionistic first-order logic formula about Peano arithmetic (Heyting arithmetic) and constructs a natural number from it, representing the (constructive) proof of that formula. Only provable formulas are realized. The point of this exercise is to prove various metatheorems about the realized language: is it consistent, and what are provable/unprovable in the intuitionistic formulae.

Intuitively, something is unprovable if there exists a formula, but there does not exist a realization of it. This can be shown by connecting the formula to the set of realizers (in this case, natural numbers), but showing that there cannot exist a related natural number (or, more often, function on natural numbers represented by its Gödel number) with the properties required of the realizability interpretation. The simplest example: since “false” is unprovable (it has no realization, by construction), the intuitionistic logic is consistent.

This also lets us prove something about the class of all provable statements. Since we have a method for constructing something from any provable (or true) statement, we can say something about the set of all provable statement in relation to the realizers. Kleene mentions one consequence is that the intuitionistic calculus cannot prove the existance of any function other than a general recursive function, since those are the only functions constructed in the realizability interpretation. This tells us, for example, that the intuitionistic calculus is different from classical set theory, which contains other functions.

An important detail in this paper that clarifies the distinction between the intuitionistic and the classical happens in Clause 6, on page 113. This is the definition of the realizability interpretation for existential quantification ∃x.A(x). This has a realization if, for some x, A(x) has a realization. It’s important to notice that this second “for some x” quantification happens in the metalanguage, namely, classical set theory, and therefore could be choosen by Choice. Kleene discusses this on page 118, where he uses the word “classically” as a modifier on various quantifiers to remind us that, when working with the quantification and realizers directly, we are working in a classical system in which intuitionistic proofs also exist.

What seems to be going on here is that the realizers are something like the intuitionistic subset of classical set theory. I think that statement isn’t exactly true; Kleene uses classical choice when working with the realizers to show there are unprovable theorems. For example, a realizer parameterized over (classically) all variables may not correspond to an intuitionistic formula. So it’s not that the realizers are only intuitionistic, I think. But any particular realizer is (must be)? The important point may be the realizers are a subset of the whole system, and thus we can prove interesting metatheorems that rely on distinguishing the realizers (and therefore, the formulae they realizer) from all the things in the full system.

amadio1998 - Domains and Lambda-Calculi, Chapter 15

Chapter 15 of Amadio and Currien’s book “Domains and Lambda-Calculus” introduces realizability in its historical context. The introduction formalizes Kleene’s work as an example, and discusses its use.

They emphasize two things, which seem to confirm some of my understanding:

The realizability relation is defined inductively over formulas, and relates formulas to proofs.
The use lets us reason about all proofs in the system.

This is the best definition of realizability I’ve seen, and applies both to Kleene’s original, but also to uses in PL.

The authors point out that Kleene’s original goal was to prove consistency. They then confirm my above intuitions, that the realizability interpretation also lets us prove metatheorems about what is provable/unprovable in the realized system. However, they note that one application of this is to find unprovable true statements, which can be consistently axiomatized back into the original system. There are proofs in the set of realizers, i.e., true statements, that are never constructed by the realizability interpretation. These could be added back to the original system to enrich it.

This latter use seems to confirm one feature of realizability that isn’t explicit stated anywhere, but seems to be true of all realizability interpretations I’ve seen: that the realizers are a strict subsystem of some larger formal system.

How is “realizability” used in PL?

In programming languages, we’re not often concerned with intuitionistic vs classical logic; we’re working constructively by default. In fact, many of the uses of “realizability” in PL don’t seem to be related to logic at all, but to modeling well-typed programs. And while, sure, these are related by Curry-Howard, the difference seems important to me. So what does realizability mean in this context?

In most uses in PL, the important feature seems to be clause 3 in my definition above: the collection of all values is larger than the set of realizers. In PL, this suggests that we’re ascribing types to “untyped” terms, and the realizers are those that are semantically well typed, but not necessarily syntactically well typed. The full collection contains also untyped terms, and we can therefore prove through realizability that the type system rules out ill-typed terms.

There do seem to be some examples in PL that are explicitly relating classical and intuitionistic ideas, namely those trying to import constructive interpretations of classic logic. I’m not really interested in those, and I think the connection to realizability is much more clear in those applications, so I’ll ignore that area.

Let’s look at some examples.

benton2010 - Realizability and Compositional Compiler Correctness for a Polymorphic Language

In “Realizability and Compositional Compiler Correctness for a Polymorphic Language”, Benton and Hur define a “realizability” interpretation of System F types realized by terms in low-level language, for proving some compiler correctness properties. The terms realize the types, and this lets us talk about which low-level programs are valid to link with, without restricting the set of linkable programs to only those generated by the compiler.

This has lost all connection to intuitionistic vs classical logic, but I suppose it keeps the key features of the technique: types (formula) of one language are realized by terms in another, and there is some concern that the realizers should be a subset of all terms. Not all low-level programs should be valid, but some set of them should be.

nakano2000 - A Modality for Recursion

“A Modality for Recursion” was actually the start of my realizability journey. This paper starts by defining a collection of models (β-models) of the untyped λ-calculus. It then defines the class of realizability models, in terms of β-models, for an extrinsically typed λ-calculus with equi-recursive types. A realizability model is parameterized by a β-model, and is a relation inductively defined over types to their realizers, which are values drawn from the β-model.

So why is this realizability? Well, I don’t see anything to do with intuitionistic vs classical. But, the set of all values is larger than the set of realizers, which seems to be important to all uses of “realizability”, and important for this result in particular. In this paper, this is used to show that the dot modality rules out some valid β-model terms, namely those that would correspond to non-terminating λ terms.

Later in the paper, they define a “realizability interpretation”. This seems to be distinct from the collection of all realizability models in that they pick a particular set of realizers? So, it ought to be a realizability model, I guess? But they don’t say so explicitly. The interpretation is still quite heavily parameterized, but it does seem to fix or restrict the set of realizers. Anyway, this interpretation includes all the features of my definition above: it’s inductively defined over types, relating types to (a semantic model of) untyped λ terms, for the purposes of proving something about the collection of realizers as they related to the collection of all untyped λ terms.

The A Means A

2022-06-30T17:25:55Z

I have argued about the definition of “ANF” many times. I have looked at the history and origins, and studied the translation, and spoken to the authors. And yet people insist I’m “quacking” because I insist that “ANF” means “A-normal form”, where the “A” only means “A”.

Here, I write down the best version of my perspective so far, so I can just point people to it.

I want to answer three question: what does the A mean, why does the A matter, and where does the A come from.

What does the A mean?

The “A” in “A-normal form” refers to a particular formal object, named “A” (not “administrative”), with respect to which there is a normal form with certain useful properties. This form is “A normal”—none of the A reductions apply to terms in this form—hence, A-normal form.

While it’s true that the history of ANF is concerned with “administrative reductions” in CPS, this is an informal concept, modeled by the formal object “A”.

In truth, “A” is several formal objects, defined somewhat differently in at least 3 different papers. Only one of these is arguably called “administrative”, but is about CPS, and not what we now call ANF.

“A” appears in “The Essence of Compiling with Continuations”, page 5. Under the discussion of the CPS, optimization, and un-CPS diagram, the authors observe that this diagram begs for a completion, some direct process, “A”, that simply normalizes a term within the same language. This diagram is reproduced below: \[ \begin{array}{ccc} e & \overset{CPS}{\to} & e' \\ \overset{A}{\downarrow} && \overset{\beta}{\downarrow} \\ e_A & \overset{unCPS}{\leftarrow} & e_O \end{array} \] They ask, what are some set of reductions, call this set A, such that normalizing with respect to A would produce a normal form, A-normal form, that characterizes the use of CPS in practice.

The same pattern appears in "Reasoning about Programs in Continuation-Passing Style, page 1:

Thus, we refine this question as follows: Is there a set of axioms, A, that extend the call-by-value λ-calculus such that: …

The authors go on to define the set A, never naming it by some administrative reductions, but deriving A instead from the inverse CPS translation.

We could argue that Sabry’s thesis, Chapter 3, Section 1, “Administrative Source Reduction: The A-Reductions”, names the A-reductions “administrative”. He goes on to analyse those reductions considered to be the administrative ones, defining βlift and βflat in terms of CPS. He then defines in Definition 3.1, the administrative source reduction (A-reductions). However, these refer to reductions over CPS terms, and are distinct from the reductions considered for ANF. While they are the origin of ANF, they do not produce terms in what we now call ANF. A term in “administrative normal form” with respect to that set of reductions would actually be in CPS. That’s not what we mean when we say ANF; we mean normal with respect to the set A defined in “The Essence of Compiling with Continuations”.

Maintaining this distinction between the formal object A and the informal notion of administrative reductions is important for two reasons. First, it helps remind us that ANF is a form ultimately about normalizing a specific set of reductions, not the output of a particular translation, which is important in practice. Implementations often relax ANF until code generation, by omitting some of the A reductions, typically, A2 in “The Essence of Compiling with Continuations”—even that paper relaxes A2 in their implementation in the appendix, because A2 leads to exponential code duplication or requires object-language continuations (“join points”). It’s hard to even formally discuss this relaxation if we do not have the set of normalized reductions in mind. Second, the idea that ANF is free of “administrative” redexes is absurd, since the idea of the administrative redex is an informal concept: a reduction that isn’t really necessary but merely an artifact of the translation. It is easy to introduce such administrative redexes in ANF; e.g., let x = y in x contains an extra unnecessary ζ redex, but it is in ANF. It is, however, free of A reductions.

Why does the A matter?

I don’t actually care what the “A” means, or what the authors intended it to mean. I care that we think about ANF as a normal form, normal with respect to a specific set of reductions.

This most recent rant was triggered by a conversation with a reviewer, who, after observing that the “A” actually stood for “administrative”, asked whether our ANF translation could be decomposed into two translations, one that did everything but normalize the ifs (handling if is annoying in ANF, as it either requires being clever or causes code duplication), and then separately handle if.

The answer is completely obvious… if you think about ANF in terms of a normal form with respect to a set of reductions, and not as merely the output of some translation process, nor “CPS but like without adminsitrative redexes”. Since ANF is a normal form with respect to “A”, we can easily decompose it into multiple normal forms, thus deriving several decomposed translations: remove the A reduction that normalizes if, and you get another normal form. Remove the rules that normalize if and nested let, and you get monadic form.

But all of this is much more complicated to explain if you think of ANF as a particular translation or particular syntactic form, and not a normal form with respect to the set A. And this seems to be very likely how you will think of you think A means administrative.

Where does the A come from??

Incidentally, I spoke with Amr after he read this blog post. The origin of the “A” comes from a result by Curry, who proves some theorems about any combinatory logic extended by a set A of ground equations: https://staff.fnwi.uva.nl/p.h.rodenburg/Varia/RelCLlam.pdf

This led Matthias to ask Amr to create a set A, such that bla bla bla.

Amr admits he may have intended a pun between A and administrative, but doesn’t remember.

What is peer reviewing?

2022-04-26T23:56:04Z

I’ve been doing, and experiencing, a lot of peer reviewing lately. I’ve been ranting about it on Twitter as I get reviews that don’t help me and, in many ways, hurt me, and lauding reviews that provide useful constructive feedback (even if I disagree with it or the decisions). I’ve been trying to figure out how to provide good reviews and avoid negative aspects of reviewing.

I need to get the thoughts out of my head. These are not declarations of what peer reviewing is or should be, but my attempt to work through those questions.

The Scientific Aspect of Reviewing🔗

scientific | adjective: based on or characterized by the methods and principles of science.

Science | noun: a systematically organized body of knowledge.

Mostly we think about the scientific aspect of peer reviewing. If we’ve written a scientific piece, we have tried to answer a question, scientifically. We have posed a research question, hypothesized some answer, tried to evaluate the answer objectively, and present all of that clearly to advance the state of knowledge.

The point of peer review, then, is to check that we have indeed done some science. To check our question is reasonable, that our evaluation is not flawed, and that we have indeed advanced the state of knowledge (and communicated that knowledge clearly, at least relatively to some community).

This interpretation of peer review is intuitive, but is it real?

Here are the evaluation criteria from the call for papers for some major programming languages conferences I’ve been involved in. They’re all SIGPLAN conferences, so they are all fairly similar:

POPL 2022:

The Review Committee will evaluate the technical contribution of each submission as well as its accessibility to both experts and the general POPL audience. All papers will be judged on significance, originality, relevance, correctness, and clarity. Each paper must explain its scientific contribution in both general and technical terms, identifying what has been accomplished, explaining why it is significant, and comparing it with previous work.
PLDI 2022:

Reviewers will evaluate each contribution for its accuracy, significance, originality, and clarity. Submissions should be organized to communicate clearly to a broad programming-language audience as well as to experts on the paper’s topics. Papers should identify what has been accomplished and how it relates to previous work.
ICFP 2022:

Submissions will be evaluated according to their relevance, correctness, significance, originality, and clarity. Each submission should explain its contributions in both general and technical terms, clearly identifying what has been accomplished, explaining why it is significant, and comparing it with previous work. The technical content should be accessible to a broad audience.

Correctness🔗

Accuracy or correctness appear in all of these. This makes sense; one major aspect of reviewing is to make sure the scientific work hasn’t made any mistakes. Of course, the reviewing itself might make mistakes. The goal isn’t to do a perfect job, but to try. And if we try, keep doing science, keep checking each others’ work, keep asking questions and even re-asking questions, eventually we’ll approach something resembling the truth. I like this criterion and think it’s probably the most important one.

Originality🔗

Originality appears in all three as well. This one is a little odd. If the point of science is to advance the state of knowledge, it makes sense that scientific work should be original, i.e., new, novel, producing something that was previously unknown. But, it also seems a little at odds with the previous criteria. One great way for checking correctness is reproducing or replicating prior results, double-checking existing work and making sure we get the same answer. The emphasis on originality or novelty seems to be at odds this goal. We could interpret it a little more generously, by considering replication and reproduction as new in the sense that they are new evaluations of an old question, so it is still original work. That’s okay with me. But it does require a little care in the interpretation of the reviewing criteria....

Clarity🔗

Clarity appears in all three as well. This is interesting as it maybe seems irrelevant to the idea of a rigorous and objective evaluation of a research question.

It may even seem.. not true of essentially any research paper. If you’ve ever tried to read one of my research papers, and you’re not literally in my field, and even for many people who are, you’d probably say they’re hard to read. Maybe not very clear, maybe lots of obtuse notation, vocabulary, methodologies, ideas, ... But the reviews have judged them to be clear enough for accepting.

So clarity seems to be quite ambiguous, quite subjective. At the very least it’s relative. Let’s say it’s relative to the community reviewing and calling for papers.

But why is it important? As long as the work is correct! Well, the point of science isn’t merely evaluating a research question, but advancing the state of knowledge. We can’t very well advance knowledge if no one, or a vast majority of a community of interest, can understand what you’ve done.

So it’s important for the work to be not only well evaluated, but clearly (relatively) well evaluated, and well communicated. This way it advance knowledge for many people, and not just the authors.

So far, so scientific. All these criteria directly relate to the original scientific goal.

Relevance🔗

POPL and ICFP include "relevance". I take this to mean relevance to the field. For example, submitting a machine learning paper to POPL is probably not relevance, even if it is of the highest quality science.

This is related to the advancement of knowledge, since only reviewers who are familiar with the area, methodologies, state of knowledge etc are going to be capable of assessing the other criteria.

This leads to some problems at the borders between areas. What if it is relevant, but heavily in another field, and as a result, there isn’t very much expertise to review the paper? But one hears stories about these papers having a hard time finding a place in any venue. I guess I should take a charitable view of such papers, but then, that may require sacrificing review quality.

Or what about work that is revolutionary, in the sense that it is creating completely new models, methodologies? Such work would argue that it is relevant, but it would be difficult for the reviewers to judge it so. It might seem completely irrelevant— no one has used it yet, perfectly good techniques (which will, by nature, be more clear and easier to judge correctness of) will apply in many of the examples.

I don’t think there’s a good solution here. Revolutions are hard. Founding new fields, subfields, etc, is hard.

Significance🔗

The last criteria for PLDI is significance, which also appears for POPL and ICFP.

I have no idea what this means. The dictionary provides:

significance | noun [mass noun]: the quality of being worthy of attention; importance

This seems like a very suspect criterion. How do we know, apriori, which ideas are important, are worthy of attention? This requires us to understand, in advance, what impact the work could have? Are we psychic?

I suppose, in some cases, it might be somewhat clear that work is not important. Sam TH, who I love for pushing against my (and others’) simple takes to advocate for complexity, recommended this paper: https://doi.org/10.1007/s11245-006-0005-2. It introduces a strawman in the context of philosphy, of people inventing and then studying a game called "chmess". One could investigate all of these questions scientifically, but they would all be insignificant, because the made up game affects no one in the world.

In this case, it’s clear that the questions are not significant, but it introduces the key problem with significance— the answer to the question is a relative one (like clarity). Maybe a question is not significant to one field, but is to another, because that field knows how to apply that kind of question to some other problem that is significant, and so on.

I don’t know how to detect significance then.

The chmess paper suggests asking whether you could explain the research question to an outside audience and convince them of its importance. This seems like a low bar, however. Almost every paper I read begins with a motivation section, which does exactly that: here’s some interesting the theoretical problem and how it could, in principle, be used to solve or address or make progress on some real world problem. Perhaps that just means we’re all good at (convincing ourselves that we’re) working on significant problems.

Supposing I could accurately judge significance by this definition, how does it relate to the original goal: ensuring that a scientific work advances knowledge? Well, if the result is unimportant, one might argue (if one were a pragmatist, in the sense of philosophy sense) that unimportant knowledge is not useful, and therefore, not knowledge. But it still seems quite difficult to judge the utility of knowledge.

It reminds me of number theory, which was considered quite without practical application. Until we invented cryptography.

While trying to understand significance on Twitter, others proposed an alternative definition: that the research question is large enough (to be important?). This seems like an even more suspect definition. For one, what is large enough? Completely subjective. The prior definition of significance is relative to a community, but this is relative to an individual’s expectations. For two, it has perverse results in practice. In an effort to ensure a result is big enough, an author is incentivized to make a result larger (or appear larger), perhaps artificially. The knowledge is withheld from the community and society until it reaches some arbitrary bar. That bar is unavoidably pushed higher, as each year new scientists joining the field compare to the current bar, new authors strive to beat that bar (from the prior incentive) and the baseline is reset.

And for what? What part of our original scientific goal is achieved by ensuring a result is "large enough"? It doesn’t help advance knowledge to withhold a result for being small, if it is correct, and clear, and original.

I see none.

At best, this definition seems to be a response to something unscientific... see the next section.

The Social Aspect of Reviewing🔗

social | adjective: relating to society or its organization.

There’s another aspect to peer review, which I will call a social aspect. Science is, of course, a social process, so these are not unrelated. But I want to separate them.

The previous aspects to reviewing dealt explicity with scientific mission— the advancement of knowledge. But it is us humans that are attempting to do that, in a large context with many systems and pressures that we interact with and within.

For example, to advnace knowledge, we must keep up with the state of knowledge. One purpose of reviewing I’ve heard advocate is to act in defense of human attention, so that we can focus on the advancement that are relevant and important. A peer reviewers job is, in part, to reject "noise" from the process, so it is even possible to know what the state of knowledge is.

This seems related to the second definition of "significance" in the prior section. If a result is "too small", perhaps it becomes a distraction. New authors spend too much time learning it and pushing it to a state of being useful, taking more time than it would have taken the original authors, thus wasting time. Or perhaps realizing the result does not scale to enough settings as originally assumed, but only after wasting a great deal of time, attention, and effort.

But this requires us to make the call on what others will want to and/or should pay attention to. That’s a hard call to make; I’m not sure whether I can make or should make it.

It could also be a reaction to gaming the system that employs scientists and funds research. A bunch of small publication is still a large publication count, which can (through the magic of relying on metrics) convince people to provide funding or jobs to one person, but not another. This, in turn, can prevent others from doing useful work to advance knowledge; they could use made use of those jobs or funding.

I’m not going to rant here about the Tyranny of Metrics, for which there’s a whole book I heartily recommend.

Metrics exist, and we have to work with them, so it’s worth paying attention to them. We don’t want people gaming our system, even if our system is stupid. We should try to change it, but that’s not always possible.

Still I think it’s worth being careful of doing more harm than good when trying to prevent people from gaming the system. I’m not sure how easy it is to recognize "salami slicing" a work in to lots of little pieces of work, apriori. Science is incremental.

On the other side, does a venue have a responsibility to game unjust systems? Some systems exist evaluate venues based on inclusion in certain indexes, acceptance reates, classificaiton as journal or conference, etc, and peoples jobs’ rely on these things. Should a venue do things like turn itself into a journal (as PACMPL has done) to game an unjust evaluation system? What about targeting a certain acceptance ratio, which POPL explicitly does?

https://www.sigplan.org/Conferences/POPL/Principles/

This acceptance ratio is important for maintaing a high ranking in the CORE ranking system.

See ICFP’s downgrade from A* to A, which cites the acceptance ratio of 33% being too high: http://portal.core.edu.au/core/media/justification/CORE2021/4612ICFP.pdf

Do reviewers have a responsibility to maintain a high ranking, to the benefit of the venue and the people submitting to their venue, by rejecting a certain proportion of papers? Without doing so, we risk the scientific endeavour— the venue and its community may no longer be able to effectively advance knowledge.

I dunno man I’m not a moral philosopher.

The Syllabus

2022-04-15T19:55:39Z

I wrote the follow at some semester prior to today, about no one in particular, while updating syllabus.

Please read the syllabus. The answer to your question is on the syllabus. If you review the syllabus you will find answers to this question and others. As I have said in many other places many times before, you can find the answer to your questions regarding course policies answered on the syllabus. The syllabus is where we describe the answer to course policies, grading, instructions on what to do in various circumstances, and answer frequently asked questions. Like this one. The syllabus is a useful tool for discovering the answer to questions about the course. Questions we have answered on the syllabus should not be asked because we have answered them on the syllabus. Piazza, email, canvas messages, office hours, after class ambushing of the professor and TAs, and lab are not the places to ask questions that are answered on the syllabus. so many times but it is not getting to me. Even if you think your situation is unusual, we have almost certainly run into it before and answered it on the syllabus. You will never make me crack. We have been teaching for decades, and get guidance from Above, and we answer the questions and put information on the syllabus. Every time you ask a question that is answered on the syllabus, the unholy child weeps the blood of virgins, and Russian hackers subtract points from your final grade. Asking questions that are answered on the syllabus summons tainted souls into the realm of the living. Questions answered on the syllabus go together like love, marriage, and ritual infanticide. The syllabus cannot hold it is too late. The force of questions already answered on the syllabus in the same conceptual space will destroy your mind like so much watery putty. If you ask questions answered on the syllabus you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. The endless questions already answered on the syllabus will liquify the nerves of the sentient whilst you observe, your psyche withering in the onslaught of horror. ((Questions-answered-on-Syll̿̔̉us are the cancer that is killing the Piazza it is too late it is too late we cannot be saved the transgression of a chi͡ld ensures syllabus will consume all living tissue (except for questions already answered there which it cannot, as they have been answered on the syllabus) dear lord help us how can anyone survive this scourge asking questions answered on the syllabus has doomed humanity to ( an eternity of dread torture and time wasted that could have spent teaching as questions already answered on the syllabus establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like Grading Scheme, but more corrupt) a mere glimpse of the world of questions answered on the syllabus will instantly transport a students’ consciousness into a world of ceaseless screaming, he comes, the pestilent slithy syllabus-infection will devour your questions, application and existence for all time like The Design Recipe only worse he comes he comes do not fight he com̡e̶s, ̕h̵is un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, syllabus questions lea͠ki̧n͘g fr̶ǫm ̡yo͟ur eye͢s̸ ̛l̕ik͏e liquid pain, the song of re̸gular waitlist will extinguish the voices of mortal man from the sphere I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful the final snuffing of the lies of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL IS LOST the pon̷y he comes he c̶̮omes he comes the ichor permeates all MY FACE MY FACE ᵒh god no NO NOO̼OO NΘ stop the an*̶͑̾̾̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e not rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ (((

)

What is the Point of a Final Exam

2022-01-07T04:11:16Z

It’s weird, but professors are almost never taught how to teach, how to design a course, how to assess students, how to design an exam or what the point of an exam even is. We’re just expected to pick this up on our own, I guess. It’s not as nonsensical as it sounds, since we are trained how to do research and communicate that research, and there is some overlap. But still.

If my experience is any indication, we just pick up an existing course structure and more or less follow that. Oh, the last person who taught this course used this material, and this syllabus, and these exams, so I’ll just do more or less that for now. If we’re ambitious and/or want to shoot our tenure track in the foot, we might try to innovate soon after. Otherwise, we might innovate later.

Anyway I’m not good at doing things just because that’s how they’ve always been done; I need first principles. After designing, administering, grading, invigilating several of them, I was struggling to figure out what the point of a final exam is. So I had a bunch of conversations on Twitter and now I’m collecting my thoughts on what the point of a final exam is and how it might, or might not, serve a purpose in my course.

Warning: I am not an education researcher, and this is not research, and it’s got a lot of stream-of-consciousness.

What is the point of a final?

Obviously, a final exam is a part of a course, so what is the point of a course? Presumably, to teach students. If only we lived in such a utopia.

I’d start by assuming the purpose of a university course is:

to teach students; and
to credential students.

I consider credentialing to be less important than teaching, but acknowledge that there is a use in credentialing, so I should do it and do it well.

Credentialing

A final exam can very easily accomplish credentialing: you put everything the students should have learned in the course on the final exam, and you measure how well they do, giving them a grade (credential).

But is a final exam the best way to accomplish credentialing? What goes into credentialing? I think the main principle is that we’d like the credential to be close to ground truth. A high mark should indicate high mastery, and vice versa. How do we know if an exam is measuring mastery well?

After a bunch of discussions, I think there are a few threats that could cause measurement error in any assessment:

cheating;
student performance error (e.g., stress, anxiety, illness);
design error (e.g., confusing instructions or questions, too little time to complete assessment, questions that don’t measure learning objectives);
attribution error (e.g., measuring a group assessment that doesn’t reflect an individual’s mastery);
sampling error (e.g., a grade on an assessment early in the semester does not actually indicate lack of mastery if the student learns later but has no opportunity to demonstrate improvement later; or, an assessment in one context may not be representative in another context, so another assessment in another context is useful).

I’m going to mostly ignore (3). It’s a research area of its own (e.g., Item Response Theory, Concept Inventory), for one, and I’m no expert in that area. But design errors in an exam are as likely (or unlikely) as in any other assessment, or perhaps less because course staff can spend more time polishing a single exam than multiple assessments.

Exams, and final exams, reduce (1) and (4) very well. Exams are typically very structured making cheating difficult compared to homework’s, projects, etc, and to properly measure a single individual. Take home or online exam don’t have this benefit, so must be designed to be more difficult, complex, or required invasive invigilation technology.

Final exams in particular mitigate (5), since they occur at the very end of the semester. Exams in general also provide an opportunity to revisit assessing several learning objectives in another, broader, perhaps integrated context.

Exams seem very susceptible to (2). Universities typically have policies in place to support accessibility and deal with illness, to help address some of (2). But many students find a single, highly weighted exam very stressful, and they may fail to perform on this one, very important assessment, despite having mastered material and demonstrated that mastery throughout the semester. On the other hand, some students find exams far less stressful, and far less time consuming, than many intermediate assessments throughout a semester. Students who find planning or long term focus challenging, or have chronic accessibility issues, might find an exam much easier than a project, for example.

So exams are pretty good for credentialing, in that they avoid sources of measurement error, as long as they’re well designed. But they’re very stressful. So if you need a credentialing assessment that reduce (1), (4), and (5), a final exam seems like a good choice, if you can find a way to reduce stress and anxiety. Maybe novel exam structures could do that; however, novel structures might cause more of (1) and (4), and be more complex to design and thus making (3) even trickier.

Some learning objectives might also be difficult to assess with other forms of assessment than exams. For example, in my project-based compilers course, some of my learning objectives don’t fit into the course-long “write a compiler” project. They are latent skills I’d like students to learn from building a compiler, but I can’t figure out how to explicitly measure that learning in the project. This could be a failure of my ability, but it suggests the structure of exams is simpler to employ.

Teaching

I consider credentialing less important than teaching, yet I covered it first because exams seem most related to credentialing.

Some of my colleagues pointed out that you can make the exam part of the teaching process. This is very counter to how I’ve seen exams.

An exam, particularly a final exam, can be an opportunity for students to reflect on all the material students have seen. They could be presented a new challenge or material that they should be able to learn from if they’ve mastered course material. They could also be given a chance for explicit transfer from one context to another, related to point (5) in the previous section about measuring skills in different contexts.

Given the stress caused by the assessment function of an exam, particularly a final exam, I’m not sure how useful this is compared to other methods of teaching. If you want to give students the opportunity to see something in a new context, why not multiple (smaller) projects, or multiple homeworks? Novel exam structures, or lower stakes can mitigate this, as discussed above, but come with design and measurement challenges. And at what point is a less stressful, novel structured “exam” really an exam, and not something else?

Another way the exam, and final exam, serve teaching is actually taking advantage of the high stakes of the exam. By giving students adequate time and material to review for the exam, they’re able to review all the material in the course in preparation for a high-stakes exam. This review is useful for learning, even if the exam itself is not.

This seems at odds with research on teaching as it relates to grades, though. I’d suggest that it’s the grade, not the exam, that causes students to review. Teaching More by Grading Less (or Differently) notes that grades motivate students perversely, lowering interest in learning but causing anxiety and interest in avoiding a bad grade. This suggests exams as motivator for reviewing material are not a good way to motivate learning.

Final exams in particular seem to next to useless for teaching since students also don’t get very much qualitative feedback. Even if they do, they have very little reason, method, or even time to review that feedback and apply it. So even if the exam is cleverly structured to enable learning in a new context, or lets students integrate disparate lessons, will they know whether they successfully integrated lessons or transfered skills to a new context? Even if the instructor spent the time to provide qualitative feedback (which, in my experience, is not what happens with a final exam), the incentives are against the student using it.

Is a final exam right for my course?

For Teaching

I’m not really convinced by the value of the exam for teaching, in itself. But, I could see two reasons:

your course doesn’t provide another final opportunity for students to integrate disparate lessons into a whole;
your course must introduce a variety of disparate skills that students have little opportunity to revisit, but that a final exam encourages them to review.

This requires mitigating stress causing performance problems on the exam, or destroying the intrinsic motivations for learning. I don’t think stress is a problem in itself, if it can be managed. I’m more concerned about replacing intrinsic motivation with extrinsic.

For Credentialing

Final exams seem very useful for courses with group work, or where detecting cheating is difficult. I’d hate to optimize for trying to catch cheating, but I think detecting it is necessary, particularly since credentialing is part of the goal of a university course. In-person exams are a much less invasive and less time-consuming mode of invigilating.

Similarly, with heavy group work, it is difficult to assess individual mastery. Alternatives include oral assessment or review of the group work. This is resource intensive, but also enables more qualitative feedback so could have additional value for teaching.

An Example

In my case, I’m trying to decide whether exams are right for my compilers course. The course is largely structured around a single semester-long project where the students implement a compiler (a big software project). The first two weeks milestones are completed individually, but the rest of the semester is completed in a group. Students who complete and contribute to the project can be reasonably assumed to have mastered a large portion of the material and most of the learning objectives. The project is not graded until the end of the semester, and students are provided considerable intermediate feedback, so have a lot of opportunity to learn from feedback, improve, and demonstrate improvement.

Cheating is a minor concern, but shirking is a bigger concern. A student could easily coast on their group.

I think an exam has some minor effect for discouraging shirking, and at least lets us catch it. I think performing oral code reviews and some form of survey of group contribution would be a more direct way of identifying this and providing more qualitative feedback. However, I don’t really have the resources for more than about 1 code review. We could perhaps implement it stochasticly.

An exam provides some opportunity for learning, in itself, but at a high cost. I can use it to have students apply the same material in a new context. However, in such a time constrained and high-stakes setting that I’m not convinced it is particularly effect or is worth the cost. The project itself is cumulative, and provides all the opportunity and material needed to review the course, so encouraging review isn’t necessary.

Compared to an exam, providing more, low-stakes opportunities to apply lessons in a new context seems like a better approach, since it mitigates (2) and provide more opportunity for feedback, learning from the feedback, and demonstrating learning. Making these exercises individual would address cheating and shirking, but might require them to be higher stakes, at least cumulatively. This would have a stronger effect for detecting shirking early. However, this would disadvantage students who find multiple assessments problematic. Such students are already at a disadvantage because of the time investment the project requires.

The exam does provide an opportunity to assess some learning objectives that I can’t figure out how to assess in the project. Perhaps I could integrate these into the aforementioned lower stakes exercises, or figure out how to integrate them into the project. Or perhaps these learning objectives aren’t really important, if I think the project is most important.

So it seems like the primary purpose the exam is serving in my course is credentialing. It helps us avoid attribution error, and to a lesser extent sampling error, at the cost of normal performance errors. It serves little, but some, learning function.

The learning function might be better served by multiple excesses outside the project. There’s a tension in using them: they need to be high stakes to serve the credentialing function and avoid sampling error, but low stakes to serve the learning functions. Sampling error is not a big problem in this course, so maybe I should favour low stakes exercises.

Attribution error is a problem, but stochastic code review and group work surveys might be a better way to address this than an exam.

If the exam is not in-person, the increased possibility of cheating negates some effect on attribution error, although code similarity detection tools make detecting cheating easier than detecting attribution error.

The exam is perhaps a simpler mechanism to employ than a combination of more small exercises and stochastic code review, but could result in more performance error.

Is an exam right? I don’t think it’s a wrong choice, but I think there are better choices particularly in favour of learning, and now I have a better idea of what the trade-offs are.

Enabling CORS for nginx WebDAV and CalDAV reverse-proxy

2021-05-13T20:08:26Z

The past few weeks I’ve been learning to develop and deploy a Progress Web App (PWA) that can communicate with my WebDAV and CalDAV servers. Unfortunately, while these are on the same domain, they are on different sub-domains, and this causes the requests to be considered cross-origin requests. For security reasons, cross-origin requests are blocked by most browsers by default unless the server explicitly allows cross-origin resource sharing (CORS). This is pretty easy to set up for static resources or scripts, if they use default headers and GET and POST methods. However, it’s particularly complicated for WebDAV, CalDAV, and other protocols that use additional headers or methods.

1 TLDR

2 CORS Requests and Responses

2.1 Preflight

2.1.1 Preflight Request

2.1.2 Preflight Response

2.2 Cross-Origin Requests

3 Configuring nginx

3.1 Configure Valid Cross-Origin Hosts

3.2 Configure CORS Headers

3.3 Process CORS Requests

4 Conclusion and Debugging

1 TLDR🔗

Copy/paste/modify the below snippets into your nginx.conf in the correct places. You’ll need to add the map declarations to http context, and merge the two server declarations into your WebDAV and CalDAV server configuration blocks. You’ll also need to customize the safelist that sets $cors_origin_header, and possibly the $cors_expose_headers and $cors_allow_headers variables.

cors-nginx.conf

http {
  # .. in http context ..
  # Declare the safe cross-origin hosts
  map $http_origin $cors_origin_header {
    hostnames;
    default "https://example.com";
    "https://example.com" "$http_origin";
    "https://www.example.com" "$http_origin";
  }
  # Declare CORS exposed response headers
  map $host $std_response_headers {
    default "Content-Type, Content-Range, Content-Language, Date, Content-Length, Content-Encoding";
  }
  map $host $cache_control_response_headers  {
    default "Etag, Last-Modified";
  }
  map $host $dav_response_headers {
    default "Dav";
  }
  map $host $cors_expose_headers {
    default "${dav_response_headers}, ${std_response_headers}, ${cache_control_response_headers}";
  }
  # Declare CORS allowed request headers
  map $host $std_request_headers {
    default "Authorization, Origin, X-Requested-With, Range, Accept-Encoding, Content-Length, Content-Type";
  }
  map $host $dav_request_headers {
    default "If-Match, If-None-Match, If-Modified-Since, Depth";
  }
  map $host $cors_allow_headers {
    default "${dav_request_headers}, ${std_request_headers}";
  }
  # Detect a preflight request
  map $http_access_control_request_headers $preflight_h {
    default "true";
    "" "false";
  }
  map $http_access_control_request_method $preflight_m {
    default "true";
    "" "false";
  }
  map $request_method $preflight {
    default "false";
    "OPTIONS" "${preflight_h}${preflight_m}true";
  }
  # Configure WebDAV
  server {
    listen       443 ssl http2;
    listen       [::]:443 ssl http2;
    server_name  webdav.example.com;

    location /.well-known/ {
      root /srv/http/www;
    }

    # Advertise CORS access controls.
    add_header "Access-Control-Allow-Origin" "$cors_origin_header" always;
    add_header "Access-Control-Allow-Credentials" "true" always;
    add_header "Access-Control-Expose-Headers" "$cors_expose_headers" always;

    location / {
      # Handle preflight request
      if ($preflight = "truetruetrue"){
         add_header "Access-Control-Allow-Origin" "$cors_origin_header";
         add_header "Access-Control-Allow-Headers" "$cors_allow_headers";
         add_header "Access-Control-Allow-Methods" "PROPFIND, COPY, MOVE, MKCOL, CONNECT, DELETE, DONE, GET, HEAD, OPTIONS, PATCH, POST, PUT";
         add_header "Access-Control-Max-Age" 1728000;
         add_header "Content-Type" "text/plain charset=UTF-8";
         add_header "Content-Length" 0;
         return 204;
      }

      auth_basic "Not currently available";
      auth_basic_user_file /etc/nginx/htpasswd;
      root /srv/http/webdav/data;
      client_body_temp_path /tmp/nginx-webdav;
      client_max_body_size 0;

      dav_methods PUT DELETE MKCOL COPY MOVE;
      dav_ext_methods PROPFIND OPTIONS;

      create_full_put_path on;
      dav_access user:rw group:r;

      autoindex on;
    }
  }

  # CalDAV and CardDAV
  server {
    listen       443 ssl http2;
    listen       [::]:443 ssl http2;
    server_name  caldav.example.com carddav.example.com;

    location /.well-known/ {
      root /srv/http/www;
    }

    location /.well-known/caldav {
      return 301 https://caldav.example.com/;
    }

    location /.well-known/carddav {
      return 301 https://carddav.example.com/;
    }

    add_header "Access-Control-Allow-Origin" "$cors_origin_header" always;
    add_header "Access-Control-Allow-Credentials" "true" always;
    add_header "Access-Control-Expose-Headers" "$cors_expose_headers" always;

    location / {
      if ($preflight = "truetruetrue"){
         add_header "Access-Control-Allow-Origin" "$cors_origin_header";
         add_header "Access-Control-Allow-Headers" "$cors_allow_headers";
         add_header "Access-Control-Allow-Methods" "REPORT, PROPFIND, COPY, MOVE, MKCOL, CONNECT, DELETE, DONE, GET, HEAD, OPTIONS, PATCH, POST, PUT";
         add_header "Access-Control-Max-Age" 1728000;
         add_header "Content-Type" "text/plain charset=UTF-8";
         add_header "Content-Length" 0;
         return 204;
      }

      auth_basic "Not currently available";
      auth_basic_user_file /etc/nginx/caldav/htpasswd;
      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_pass_header Authorization;
      proxy_pass        http://127.0.0.1:5232/;
    }
  }
}

2 CORS Requests and Responses🔗

2.1 Preflight🔗

When a script running in a secure browser attempts to make a cross-origin request, the browser first send a preflight request (for non-trivial requests), and then sends the actual request if the server advertises that CORS is enabled for that request. A preflight request might be skipped for an HTTP GET method request, because this is considered harmless.

2.1.1 Preflight Request🔗

Essentially, a preflight request is the browser asking the server for permission to make a request of a certain METHOD and then share the data and certain headers with a third party. The preflight request is an HTTP OPTIONS method request with the following headers set:

Access-Control-Request-Headers, which declares the headers that the cross-origin script is requesting.
Access-Control-Request-Method, which declares the method of the request that the cross-origin script wants to send.
Origin, which declares the domain of the origin of the script that wants to make a cross-origin request

For HTTP servers serving static content or scripts that don’t use OPTIONS, it’s enough to detect an OPTIONS request, set the above headers, and return a 204 status code. For some HTTP servers, like WebDAV and CalDAV, the OPTIONS request has another use, and we really have to detect a preflight reuqest by detecting both an OPTIONS request and the preflight request headers.

2.1.2 Preflight Response🔗

To respond to a preflight request, the server is expected to reply with an empty content response, HTTP status code 204, and the following headers:

Access-Control-Allow-Origin, which declares the hostnames that are allowed to make cross-origin requests. This ought to include the Origin of preflight request for the preflight request to succeed, and can be a wildcard value *.
Access-Control-Allow-Headers, which declares which headers are allowed to be part of the cross-origin request. These are all be HTTP request headers.
Access-Control-Allow-Methods, which declares which HTTP methods are allowed as part of cross-origin request.
Access-Control-Max-Age, an optional header that declares how long this response to a preflight request can be cached;

The 204 status code declares a success with no content. An HTTP request header is one that originates from the client and is part of a request from the client.

See https://developer.mozilla.org/en-US/docs/Glossary/Request_header for more.

Some browsers (such as Firefox, and Chromium) will consider the preflight request as succeeding if the above headers are present, even if the status code is not 204, and even if the request contains other data.

Sme headers are part of the Access-Control-Allow-Headers by default, as they are considered safe.

See https://developer.mozilla.org/en-US/docs/Glossary/CORS-safelisted_response_header for more details.

Figuring out exactly which headers to list in Access-Control-Allow-Headers is a little annoying. For my WebDAV (nginx) and CalDAV (radicale) servers, the following list seemed sufficient for my uses:

The following standard headers: Authorization, Origin, X-Requested-With, Range, Accept-Encoding, Content-Length, Content-Type;
The following DAV headers: If-Match, If-None-Match, If-Modified-Since, Depth

This will depend on exactly what web app is communicating with the server and what it relies on, what the underlying server is. You may need to do a bunch of testing in the web developer’s console to figure it out.

Similarly, figuring out exactly which methods to list in Access-Control-Allow-Methods depends on the app and server (but not the browser). These methods are probably more well-specified. For WebDAV and CalDAV, the following were sufficient: REPORT, PROPFIND, COPY, MOVE, MKCOL, CONNECT, DELETE, DONE, GET, HEAD, OPTIONS, PATCH, POST, PUT.

2.2 Cross-Origin Requests🔗

After a preflight request, the browser will start sending cross-origin HTTP requests. These will be normal HTTP requests, but the browser will expect the following additional headers in the response:

Access-Control-Allow-Origin, which declares the hostnames of cross-origin scripts that this response can be shared with.
Access-Control-Allow-Credentials, which is either "true" or "false", and declares whether the authorization information in this response can be shared with cross-origin scripts.
Access-Control-Expose-Headers, which declares which HTTP response headers can be exposed to the cross-origin script.

An HTTP response header is one that originates from the server and is part of a response from the server.

See https://developer.mozilla.org/en-US/docs/Glossary/Response_header for more.

For my WebDAV and CalDAV servers, I needed to expose via Access-Control-Expose-Headers the following for my uses:

The following standard response headers: Content-Type, Content-Range, Content-Language, Date, Content-Length, Content-Encoding;
The following response headers that have to do with cache control: Etag, Last-Modified. You may want to add Pragma if you support HTTP 1.0, and Cache-Control and Expires if your server needs to direct your app about cache expiration. Etag and Last-Modified were sufficient for detecting changes between the local and remote versions of DAV files in my app.
The following DAV-specific headers: DAV.

3 Configuring nginx🔗

Configuring nginx correctly is tricky due to the design of the nginx configuration language. It is a declarative language, but can look imperative and trip us up. We have to be careful in how we conditionally add headers and process requests.

nginx also doesn’t allow us to use set to create variables in all contexts, so we have to be a little clever at times.

3.1 Configure Valid Cross-Origin Hosts🔗

To limit which domains can issue a cross-origin request, we create a safelist and set a variable based on the Origin header of the request. We use map to declare the variable $cors_origin_header to be the origin, if the origin is on the safelist.

See http://nginx.org/en/docs/http/ngx_http_map_module.html#map for more.

map $http_origin $cors_origin_header {
  hostnames;
  default "https://example.com";
  "https://example.com" "$http_origin";
  "https://www.example.com" "$http_origin";
}

In this safelist, we allow cross-origin requests from https://examples.com and https://www.examples.com, but no other hosts. We could use the wildcard "*" to allow requests from anyone.

3.2 Configure CORS Headers🔗

In http context, I use the following maps to declares the CORS request and response headers. This is an abuse of map to give us the ability do define variable in http context, since set doesn’t work in http context.

You’re free to inline these header values later, but separating them out into these variables made them easier to reuse in both the WebDAV and CalDAV servers.

# Declare allowed CORS Expose Headers; each is an HTTP response header.
map $host $std_response_headers {
  default "Content-Type, Content-Range, Content-Language, Date, Content-Length, Content-Encoding";
}
map $host $cache_control_response_headers  {
  default "Etag, Last-Modified";
}
map $host $dav_response_headers {
  default "DAV";
}
map $host $cors_expose_headers {
default "${dav_response_headers}, ${std_response_headers}, ${cache_control_response_headers}";
}

# Declare allowed CORS Request Headers; each is an http request header.
map $host $std_request_headers {
  default "Authorization, Origin, X-Requested-With, Range, Accept-Encoding, Content-Length, Content-Type";
}
map $host $dav_request_headers {
  default "If-Match, If-None-Match, If-Modified-Since, Depth";
}
map $host $cors_allow_headers {
  default "${dav_request_headers}, ${std_request_headers}";
}

3.3 Process CORS Requests🔗

Next, we need to detect a preflight request. We might be tempted to use if, but remember: If is Evil, so we want to avoid it.

Instead, we’re going to use map to create a variable that is equal to "truetruetrue" if and only if we detect a preflight request. This time, we’re using map as intended, to conditionally define variables.

map $http_origin $cors_origin_header {
  hostnames;
  default "https://example.com";
  "https://example.com" "$http_origin";
  "https://www.example.com" "$http_origin";
}

map $http_access_control_request_headers $preflight_h {
  default "true";
  "" "false";
}
map $http_access_control_request_method $preflight_m {
  default "true";
  "" "false";
}
map $request_method $preflight {
  default "false";
  "OPTIONS" "${preflight_h}${preflight_m}true";
}

We set the value of $preflight to "truetruetrue" when we detect a (non-empty) Access-Control-Request-Headers header, a (non-empty) Access-Control-Request-Method, and the request method is OPTIONS. We set the variables through string concatination to emulate boolean and, since nginx does not support nested conditions or boolean arithmetic.

To actually detect and process a preflight request, we add the following code in location context in the server on which you want to enable CORS. I add it in the location / block of both my WebDAV and CalDAV server blocks.

if ($preflight = "truetruetrue"){
   add_header "Access-Control-Allow-Origin" "$cors_origin_header";
   add_header "Access-Control-Allow-Headers" "$cors_allow_headers";
   add_header "Access-Control-Allow-Methods" "REPORT, PROPFIND, COPY, MOVE, MKCOL, CONNECT, DELETE, DONE, GET, HEAD, OPTIONS, PATCH, POST, PUT";
   add_header "Access-Control-Max-Age" 1728000;
   add_header "Content-Type" "text/plain charset=UTF-8";
   add_header "Content-Length" 0;
   return 204;
}

Note that due to limitations on add_header, this if block must appear in location context. Note that we also cannot move any add_header command outside the if. The add_header commands are not executed in a sequential order, but all of them are "executed" simultaneously as a block at the current level.

See http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header.

Note also that this if must end in return 204. This is part of the preflight request response (although some browsers will let you get away without it), and necessary for if to behave correctly, since If is Evil.

You can customize the Access-Control-Allow-Methods header depending on the server and your app to provide the least privilege.

Finally, we add the headers for other cross-origin requests. We add the following in any valid context, except the if body for the preflight request. I added them in server context.

add_header "Access-Control-Allow-Origin" "$cors_origin_header" always;
add_header "Access-Control-Allow-Credentials" "true" always;
add_header "Access-Control-Expose-Headers" "$cors_expose_headers" always;

Note that the always argument is required for non-preflight requests, since the HTTP response codes for successful requests will be variously 207, 200, and 304 (maybe others), and the add_header does not actually add a header for responses with some of these status codes.

See http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header for more details.

4 Conclusion and Debugging🔗

Now, if you look in the Network Monitor of your browser (Ctrl+Shift+E), and click "XHR", you should see some successful cross-origin requests from your web app. If you see they’re being rejected, try anaylzing the request, and changing the above configurations with additional headers or safelisted origins.

A Suitable Cutlery Tray

2021-01-18T17:59:36Z

This post is a transcription of a thread that happened live on Twitter on June 30, 2019, in response to some anger at my lack of a cutlery tray. I was in a mood following a previous cutlery incident.

IT NEEDS A TRAY

You’re right. The current state of this drawer is unacceptable. Sure, I can grab a knife, spoon, or fork with relative ease, and the rubberized mat makes cleaning the drawer simple. But look at those irregular lines. There is no order. This must be fixed!

The problem I have is in finding a suitable tray. I have, on occasion, been called obsessive, but really I just live my life in the way that make me happy. I like each item that I own to meet that goal. I don’t just want a utilitarian cutlery tray; I want one that will spark joy.

When I look at cutlery trays, the first criteria, obviously, is a perfect fit within the drawer. It must fit flushl edge-to-edge, at least to the left and right edges, and against the bottom edge. Against the top edge would be ideal, but I suppose I can live with extra top space.

The height is also a concern. The side walls of the tray and of the dividers must come exactly to the height of the drawer. This, I’m sure, is obvious to you all.

Next, the tray slots must divided evenly amongst the spoons, forks, and knives (in that order, of course, for obvious reasons). Any drawer of cutlery would have equal numbers of each, and it would be unjust to have a tray inequality divided.

At this point, most cutlery trays are already out of the running, and yet I have more (obvious and sensible) requirements. The tray must be made of wood. Plastic is abhorrent. You would not believe the difficulty in locating a wooden cutlery tray by itself.

But one that meets all the other requirements? Impossible. I’ve looked. I’ve been to Amazon, to Bed Bath and Beyond, to TJ Max. They simply don’t exist. It’s like a conspiracy. My only recourse, I fear, is to design my own.

I could, I suppose, start a company, whose sole purpose is to design the perfect cutlery tray. I could seek out designers and material engineers, source responsibly forested wood. Teak, ideally. I could find a way to construct it without the use of glue or staples or nails.

This would probably not be sufficiently profitable to stay in business long, and I’m likely to move soon and need a new design. To keep the business afloat, I should diversify. Military contracting is always profitable; perhaps I could re-purpose my engineers.

My team, experienced in perfect design, responsible materials and supply chain management, would be suited to completely dominate this new market. Soon, I would hold all military contracts, patents for unimaginable weapons—because no one before had dared imagine.

Using the proceeds of my new monopoly, I would quietly return to the cutlery market, taking over or putting out of business any company who dared manufacture “cutlery trays”… an insult to the very idea…

This, I fear, would not satisfy me, though. Obviously, every cutlery drawer should come equipped with the perfect cutlery tray. I would have to move in to the interior design market. All drawers and cabinets would be manufactured according to my perfect ideal, my grand design.

As I begin to corner this new market, the FTC becomes concerned at the grand and beautiful hegemony I bring to each new market, but I cannot let them stop me. I perform my biggest hostile take over yet: the United States of America. I naturally held on to my best weapons.

I rename it to the Grand Hegemony of Perfect Cutlery Design, and shift the focus of the economy into the design and production of cutlery trays that suite my tastes. This destroys several Chinese companies dedicated to mass production of plastic trays, and sparks a trade war.

As China is my main supplier of teak, this is a true threat to the Grand Hegemony. I am forced in to military action.

While I expect a swift victory due to my superior design in weapons, I did not count on the effectiveness of Chinese corporate espionage, nor on the retaliatory strikes from Russia and North Korea. The nuclear fallout forces the remains of my government into underground bunkers.

We eventually win. We will have complete control of the teak supply. Our cutlery tray empire is secure. The nuclear fallout will clear eventually. Until then, I have the perfect cutlery tray built in to my cutlery drawer to keep me at peace.