Introducing our Blog

Introduction

Architecture

The architecture for our blog is built to allow for multiple types of blog posts, including research papers, changelogs, and any sort of informational blog post.

Distill

The Distill templating framework was used heavily in our blog implementation.

Distill's framework provides several custom components for setting up a post, including displaying standard metadata (like authors and publication date), a table of contents, and bibliography utilities.

This built-in functionality made it perfect to just plug in directly into our existing Next.js architecture. With this, it was simple to wrap Distill's components into our own custom JSX components. Now having all of the components necessary to set up a blog post, we were able to make each blog post as its own Next.js page, with all the styling and logic already handled by Distill's framework. While some issues were encountered with conflicts between Next.js rendering and how Distill does its rendering of components, we were able to iron this out and make sure Distill's framework cooperated with our site architecture.

LaTeX Conversion

With many research papers made using the very popular LaTeX typesetting system, it was important to have a way to easily go from a paper in LaTeX form to a properly-formatted Next.js page. This required some sort of way to parse the LaTeX and eventually convert it to JSX (with our custom components used when applicable). We didn't want to reinvent the wheel and completely implement a LaTeX framework from scratch, so a few other alternatives were considered.

First, we considered simply using LLMs to do the conversion. We pasted in the LaTeX text with some information about how the conversion should be done, and asked for the output in JSX with all custom components handled. This worked quite well in many cases, but unfortunately was inconsistent at times and couldn't be relied on to do a proper one-to-one conversion due to AI hallucinations.

Instead, we landed on building out a very simple function that would use regular expressions to do rudimentary parsing of LaTeX and do one-to-one conversion to JSX. By using regular expressions, we didn't have to build a complex LaTeX parser that handled the tons of edge cases that could come up. Instead, we just did a simple replacement algorithm that would replace LaTeX commands with our components. For example, something like \section{Introduction} would get replaced with<SectionHeader id="introduction">Introduction</SectionHeader>.

This solution worked perfectly. Since it did conversion through regular expression replacement, we could rely on it being a true one-to-one conversion, unlike with the possible hallucinations of LLMs.

Bibliographies

One of the most important features for research papers in particular was a way to display a bibliography with a list of references used in the paper, and a way to have inline citations in the paper itself that links to these bibliography entries.

With many research papers made using the very popular LaTeX typesetting system, it was important to have an easy way to go from a bibliography in BibTeX format to a JSON format that could be easily displayed at the end of a blog post. The Distill framework had several functions for this, where you could import a BibTeX file and then a references list would automatically be generated at the end of the post. In addition, Distill makes it really easy to have inline citations, as you can link directly to the references you already imported with the BibTeX file.

However, Distill's pre-rendering process, where it has to completely parse the bibliography and then insert the corresponding elements into the page, kept running into conflicts with Next.js's hydration process, preventing the bibliography from being properly displayed. To avoid these conflicts and inconsistencies, we made our own custom elements to allow for the same bibliography features.

First, we borrowed Distill's code for parsing BibTeX into JSON and refactored it into functions we could easily incorporate into our React components. This made it as easy as possible to convert BibTeX text (or a BibTeX file path) to the JSON format we needed.

Once converted into JSON, it's relatively easy to use the bibliography information to display citations. A "References" section is automatically added at the end of the article, with a full ordered list of all the bibliography entries. In addition, whenever there is a part of the article that references a source, our custom Cite component can be used. This component takes in a "key" to the relevant citation, and then searches through our bibliography JSON to find the citation information matching to that key.

With the information found, the component then acts as an inline citation, showing what number the citation is in the overall bibliography. In addition, the inline citation has an interactive feature in the form of a info box that appears above the citation when a user hovers over it. Here's an example![1]

Using these custom components together, our blog can display full-featured bibliographies and citations, better supporting our use case of displaying research papers.

Workflow

To close out our architecture overview, we wanted to show what the general workflow would look like for going from a LaTeX research paper to a page on this blog.

First, you commonly would start with a folder of LaTeX files, each one representing a different section of a research paper. In addition, as with one paper we converted, there may be a "variables.tex" file that defines some values used in multiple sections. A script can then be run that takes this folder as an input, and converts each section to its own React component, with our custom styling, citation handling, and variable replacement already applied. Then, the only manual work that has to be done is filling in basic metadata for the paper (like title, authors, date, DOI) and putting the sections in order.

Without a workflow automatically handling these steps, getting a research paper ready for display on our site would be a tedious task requiring lots of time spent doing the busywork of copying things over when so much time has already been spent writing out the paper in the first place. Having utilities that do this conversion for us makes this whole process a lot more simple, and makes it a lot more feasible to convert large research papers into our blog post format for display on our site.

Conclusion

We look forward to publishing more and more various types of posts on this blog, and we hope you join us on our journey!