Magic Markup: Maintaining Document-External Markup with an LLM

Abstract

Text documents, including programs, typically have human-readable semantic structure. Historically, programmatic access to these semantics has required explicit in-document tagging. Especially in systems where the text has an execution semantics, this means it is an opt-in feature that is hard to support properly. Today, language models offer a new method: metadata can be bound to entities in changing text using a model’s human-like understanding of semantics, with no requirements on the document structure. This method expands the applications of document annotation, a fundamental operation in program writing, debugging, maintenance, and presentation. We contribute a system that employs an intelligent agent to re-tag modified programs, enabling rich annotations to automatically follow code as it evolves. We also contribute a formal problem definition, an empirical synthetic benchmark suite, and our benchmark generator. Our system achieves an accuracy of 90% on our benchmarks and can replace a document’s tags in parallel at a rate of 5 seconds per tag. While there remains significant room for improvement, we find performance reliable enough to justify further exploration of applications.

BibTeX

@inproceedings{2024-programming-magicmarkup,
  author    = {Misback, Edward and Tatlock, Zachary and Tanimoto, Steven L.},
  title     = {Magic Markup: Maintaining Document-External Markup with an LLM},
  year      = {2024},
  publisher = {Association for Computing Machinery},
  doi       = {10.1145/3660829.3660836},
  booktitle = {Companion Proceedings of the 8th International Conference on the Art, Science, and Engineering of Programming},
  series    = {<Programming> Companion '24}
}

📝 publications index