Vision

Under the guidance of Dr. Ken Arnold, the Thoughtful AI lab conducts research into applications of large language models (LLMs) which enhance—but do not replace—the users’ writing. Our lab focuses on two categories of these applications: AI-generated feedback for the revision stage of writing and AI-generated recommendations for the drafting stage of writing.

From 2023-2024, we developed a Microsoft Word add-in whose AI-generated suggestions were primed for modification and personalization, facilitating more fine-grained involvement in the writing process. This year, we aim to further this work in four ways: publishing an analysis of prior user studies on our system, utilizing the report’s findings to improve user experience, designing a longitudinal study with the improved system, and integrating revision/drafting features into a single app published on the Microsoft Store.

For more context on our project, you can read our original project proposal here.

Approaches

We developed a Microsoft Word sidebar that offered next-sentence suggestions expressed in four different ways:

• Completion of a current thought: provides a single pre-written sentence
• A question about current/future writing: a question the writer could answer in the next sentence
• A list of possible vocabulary: vocabulary to use in the next sentence
• A rhetorical strategy to employ: directions on the kind of sentence to write, such as citing statistics, offering a personal anecdote, incorporating a counterargument, etc.

Progress So Far

After conducting a pilot study with N=9 participants, we coded each of the resulting transcript logs to categorize user feedback by topic and theme. Generally, participants found questions and rhetorical moves to be useful and friendly, and found abstract prompts more useful than individual words or turns of phrase. We intend to further contextualize these findings in process-oriented writing theory to evaluate how suggestions might play a role in various stages of the writing process. Current analysis suggests that writers can gain the most from AI assistance during the pre-translation phase—that is, before they put an idea into words.

In tandem with this research, project development has continued with a focus on integrating previous works, finalizing DevOps, and improving the robustness of our system. All prior functionality is now merged into a single Microsoft Word add-in, and its backend migrated onto Azure Services for better scalability and reliability. We also implemented authentication (OAuth) to allow users to sign in to our add-in using their third-party accounts. Currently, we are in the process of publishing our application to the Microsoft Store.

Future Work