Anki Watching my Screen

As written in my previous post, I have grown to use Anki quite a lot in the past months. I built an Anki app for my watch, integrated it within ChatGPT and even made a screen overlay using AutoHotKey to facilitate the process of taking notes, reviewing flashcards and saving articles I read. However, one problem remains.

The Issue with Friction

Very cliche, but one lesson from the book Atomic Habits that always comes back to me is that about friction. While I certainly do not do this every day, I often try to set up everything I need for my following day right before I go to sleep. If I know I need to run or head straight to school, I may leave my clothes by the door or my coffee ready in the fridge. The same principle about reducing friction applies to everything, including Anki.

Preliminary Steps

While the Anki on my watch and the other extensions certainly help, one point of friction is adding new cards but more importantly, recalling what I have been working on in order to make such cards. There are many times where it is too cumbersome to stop reading a passage or reading programming docs to create a new card about it, even when I know remembering that topic may come in handy in the future. What if you could have Anki just watch your screen and reason about what you are reading, how that relates to your existing body of knowledge (so it does not propose trivial concepts) and maybe some context on who the user is.

The Watcher

Having "something" watch your screen would not be very secure, unless you are intentional about it. Say I know I am about to start work or studying for the day and I want Anki to watch only certain parts of my screen, or only for a set amount of time. Because this was something I had been thinking about for a while, I sought to tackle the problem with my friends during NexHacks at Carnegie Mellon back in January.

Architecture

In this project, the user connects to a dockerized browser through NoVNC, while a browseruse web-agent connects to the same browser through Chrome DevTools Protocol. We used Overshoot, one of the hackathon sponsors, for "always-on" screen streaming, but the same can be achieved with screenshots every few seconds or even DOM analysis -- cheaper, but not perfect if you are watching a video. Since the agent and the user share the browser, you could also ask it to perform actions for you, but that is a bonus instead of the main rationale for this.

The Rationale

Beyond having the agent take little snapshots of what you are doing in a way that does not breach the privacy of your computer -- unlike other alternatives that may perform actions on your actual desktop -- I found that this is a great way to address the distraction problem. As you have certainly experienced, it takes a considerable amount of time to regain focus after seeing an email pop up, your phone ringing or being called by your friend.

While this approach cannot handle the latter two, it can remove any distractions from your default machine, as all the work you do for that session is containerized in place.

Session Review

When you are done studying, you end the session and the system presents you with four review modes. It sends your study session data -- concepts extracted, timelogs, screenshots -- through an LLM to generate structured learning materials. You can then quiz yourself, flip through flashcards, explore a knowledge graph of connected concepts, or just chat about what you learned.

The quiz is generated based on what you were actually reading. Here is a sample from one of our sessions:

The knowledge graph is persistent across sessions -- every time you study, new concepts get added and connected to what you already know. Over time, it builds into a visual map of your learning.

Final Notes

You can export the generated flashcards directly to Anki through the same API bridge from the previous post, which means they automatically sync to the watch too. I have not revisited this project since NexHacks, though all the code is open-sourced on GitHub for those interested.