— programming, life logging — 9 min read
This post is to report my continued effort into building tools to model, analyse and organize my everyday life. I started from a really convoluted TODO app, and now I am evolving from that to track everything about myself 24/7.
After having recently settled down with a full-time job (Facebook), my life transitioned from a "swim-or-sink" mode to a "relax-and-think" mode. Since then I noticed how my human brain is not optimized for analytical tracking and modelling, whereas for social interactions. I felt the urge to create some sort of tooling to analytically model my lifestyle and provide insight on how to iteratively improve my life experience.
I worked on something similar for the last year, building my TODAG, that stands for TODO-DAG: a non-trivial graph representation of the different task that I need to do day by day. I used it for a long time and that gave a lot of time to reflect (here some notes I wrote) on how best ideate some tooling to help me improve my lifestyle on the long term. I will update here for the scope and areas of interest of the project.
A big takeaway from how I use TODAG is that having a computational model of your activities helps on different fronts, both querying "what I need to do" but also analyzing your activities objectively. This became evident when I added mood tracking to my TODAG. It allowed having a daily report on my mood correlated with my productivity, and that was really great, both in terms of analysing and also predicting my future mood in a specific situation. I see modelling as an extension of the mood tracker, here is a definition (that I will update over time).
What is lifestyle modelling? The activity of creating a computational model of my day-to-day activities, from sleeping to eating, from working to relaxing. The computational model is brought to life by tracking different metrics like sleep hours, mood, location, heart rate and so on.
If you ever worked for a big tech company (like where I work now), you see how everything is based on metrics. I find really compelling walking around floors and seeing these big screens with team health metrics broadcasted live, the main goal of most people in the company is to move the metrics (in Facebook we have stuff like user engagement, Monthly Active People, and so on). I find this metrics framework to be incredibly exciting and also pretty much well established, and I think that applying this to lifestyle modelling can lead to interesting results. Here comes the second important question, what do I do with these metrics.
Why lifestyle modelling? Having a bird-eye view on your life situation enables to take ownership easier on the different aspects of your everyday life. From goal-tracking to things like "I want to be 10% more relaxed in the next month". This means practically that you recurrently sit down and examine your metric, ideally every night. This is like in some strategy video-game match where you examine how much soldiers you lost/killed in the battle, here you sit down and examine what did you do today, and how your metrics were.
For now, this is more of a brainstorming session. I will report here my attempt to a different kind of tracking and see how they work out.
This listed will be updated with new ideas I will have about the tracking, in the meanwhile, I will start implementing something on GitHub while I will leave this blog post as a concept page for the project.
As a Google Cloud enthusiast I found a product that looks promising: Stackdriver (NOTE: I have no affiliation with that). Is basically a monitoring service from implemented with the Google Cloud suite that I am already pretty familiar using. I examined a couple of options and would I like about Stackdriver is that let you build custom metrics and then export your metrics chart in an external dashboard. The plan as follows.
A first prototype of my data monitoring web app, bootstrap template + Google Cloud backend took less than 24 hours to build it.
I have proceeded building this monitoring app and created a repo on my GitHub called datamonitor. Is basically a GAE (Google App Engine) instance with some login features that receives POST requests from logged user with metrics, it send the metrics to Google Stackdriver (sort of Prometheus like service) and then display the historical chart of all the metrics. Now next step is to start adding metrics and creating more complex queries over them. Next metric I am planning to add:
I also started to log the metrics on a POSTGRESQL database so that I can build custom chart.js diagrams on my dashboard. I recently discovered that the library is nicely integrated with my bootstrap template so I might as well give it a try.
What will I do with custom charts? A good product strategy for what I want to build is seeing your daily life as a video game. Having a dashboard with real-life metric powered custom chart enables the gamification to a super immersive extent: this is how it may look like.
7-Days-Productivity histogram (custom-chart) built using chart.js library and google stack driver backend
Now I need to figure out more detail on how to build the single custom charts. Some ideas are to have a daily aggregate with productivity and activity hour by hour during the day. The fundamental metrics are time logged for productivity tasks, the number of productivity tasks seen, mood during the day, browsing activity, and a location. I think having this 5 metrics down aggregated by hour gives you a nice holistic view on how the day went.
(01.07.2019) I started doing some research on how to log browsing activity and turns out I need to write myself a scraper since there are now public APIs to access that. I found some similar repo implemented in go , I will probably implement my own Python version of that. In the meanwhile, I improved the logging in terms of productivity tracking: I implemented a planning tool that allows me to plan the day ahead and to track exactly how much time I worked on the things I planned I was going to work on. It looks like this.
Some cards inside the Daily-recap dashboard, related to the planning tracking tool.
This fits into the daily recap dashboard, where beside planning I want to put an hour-by-hour aggregate of my daily activity, both of productivity and also of browsing history.
(09.07.2019) I used this dashboard for a week and it does its job but it looks a bit empty, I worked in previous days about integrating myactivity.google.com inside my dashboard. I am really fascinated by the amount of personal data I can find on myactivity, they will totally make my dashboard more rich and accurate. I opened a new standalone repo to implement this scarper at github.com/SolbiatiAlessandro/google_myactivity_scarper. I looked for couple of options and how to build this and I decided to use bs4 for the actual scraping and pyppetterr to run Chrome Headless. Where bs4 was kind of an obvious choice, I was not too familiar with headless browsers: you basically run chrome without the browser rendering. There is a nice library maintained by the Chrome/Chromium dudes called Puppeteer, you can run headless Chrome as a nodejs application, and the trick is that it can track your cookie sessions and automatically login into your google account to get your activity.
You can read your own activity at myactivity.google.com. I was able to build a parser that sends all my personal info to my datamonitor backend
(14.07.2019) After a lot of trials and error I was able to build a parser for Google MyActivity. Turns out the python version of the app couldn't handle persistent login session so I needed to switch to NodeJs. Now that I am parsing all my activities from Google I need to think of an efficient way to organize them. The idea I am having is to embed Google Calendar inside my web app and use the google calendar API to add them to the calendar so I can clearly see how I am spending my time. On top of that, I will add a couple of additional metrics like 'time spent browsing' and so o.
Prototype for the integration of Google My Activity and Productivity Metrics with Google Calendar (these events are being automatically generated with Google Calendar APIs)
(28.07.2019) So I started using this integration with google calendar and is a really nice idea. Is still a bit confusing since I am logging too many things now, and it looks way more confusing then the image above. What looks neat is this concept of review the day at 21:00pm. This is to make sure that you don't get carried away by things during your day. Find time to focus and think. Make checkpoints where I analyse if I am going in the right directions: don't mistake motion for progress. This review time is when you actually open Google Calendar and you check what you did during the day, and you write some ideally notes. I will try to implement this for the next few weeks and check how it works