30 Jun 2016
I’ve been increasingly interested in using Reporter as the centerpiece of a “lifelogging” strategy for some time now (for example), but arguably it’s trademark feature—random snapshots—was also the thing I found the most problematic. It always seemed like I was getting surveyed at the “wrong” time, in the middle of some trivial task, while the larger, more “important” events in my day were going uncatalogued. I know I was missing the point, but I just don’t have the patience to wait an entire year to aggregate the data and see trends.
In fact, what I really want (perhaps what I’ve always wanted) is a really great, roughly accurate snapshot of my day—time tracking on the largest scale: no billable hours, no team collaboration, just a categorized overview of how I spend every second of every day.
The problem, as I saw it, was two-fold. The first issue was computational: how to measure the time between Reporter’s reports? The second issue, though, was behavioral: could I consistently use Reporter proactively, as a kind of diary? I’m happy to report that the answer to these questions is “using time tracking software” and “yes,” respectively. Now I just make quick reports throughout the day and end up with nice pie charts. Here’s what I did yesterday:
That pie chart is generated by Toggl, which has become my time tracker of choice. It works across devices, generates detailed reports, has a robust API (and an even easier CLI), has a native menu bar app, and is free for those of us who aren’t using it for team-based work purposes.
So, here’s how this works: every new “report” from Reporter starts a Toggl timer. The next report stops the current timer and starts a new one. Repeat as needed.
Super important caveat: this uses a one-word tag from a Reporter question as the Toggl “project.” That project must already exist in Toggl, via the web, desktop, or mobile app. The Toggl CLI that we’ll be using cannot create new projects on the fly.
- Software: Reporter, Hazel
- Reporter should be set to export to Dropbox
- You should have at least two questions in your reports
- What are you doing?
- Add a project tag?
- Hazel should have a rule to check for recently created files in
~/Dropbox/Apps/Reporter-App. (Reporter recreates its export file time every time, so I found that checking for “updated” files didn’t always yield the desired results.)
- Download and set up the Toggl CLI. You can put this wherever you want, but the following script looks for it in
- Hardware: I have Hazel running on an always-on home server. If you don’t have an always-on machine somewhere, be prepared to miss some data. This just won’t work when your laptop lid is closed.
Set Hazel to run this, being sure to add your home folder name on line 14:
I never really have a sense of whether my posts are way too detailed or not detailed enough, so let me know if any of this makes sense in the comments.
12 May 2016
This is the third part in a series about my 2016 writing workflow. See part 1 here and part 2 here.
By far the biggest revelation for my plain text writing workflow has been
git. Git is a powerful command line utility for version control, perhaps best known as the basis for the code-sharing platform GitHub — though you don’t need to use GitHub to benefit from a git workflow. For our purposes, git’s strength is its ability to keep track of and compare multiple drafts of a project. If you have ever had a folder that includes files like “essay_version1,” “essay_version2,” etc., then you already know the basics of version control. Git handles that process for you, making it both more seamless and more effective.
If you already know git, you can skip down to my implementation. If you’re a total newbie, then feel free to read on.
Git works best when you keep your writing projects divided into project folders. Back when I discovered the “alternate” version of Notational Velocity, I went nuts and decided all of my documents should exist in a single folder. But once I discovered git (and stopped worrying that iOS meant the death of the filesystem), I reverted back to using folders – and embraced their skeuomorphism. Anyway, once you have that project folder, you’re ready to start using git.
Git can be as complex and you want to make it, and computer programmers have their own complicated thoughts on the best ways to work with it. But since we’re using git for writing, we can stick to the basics. Here are some key terms:
- Repo: git projects (in our case, project folders) are called repositories, or “repos” for short
- Branches: each repo can contain multiple “branches,” separate threads of the project that allow you to work on something without worrying that you will ruin what you already have. While this applies mainly to coding, it can also be an important tool for writing.
- Commit: When you commit something to git, you’re taking a snapshot of everything in the folder as it is right at that moment. You also leave a little note, called a commit message, that helps you remember what’s new or what’s changed.
- Diff(tool): to “diff” in git means to compare the differences between two different versions of your project. This can be between one commit and another (say, yesterday’s vs. today’s) or between one branch and another.
To start working in git, you need to use the command line and get into your folder:
cd path/to/folder. Once you’re there, make the folder a git repo:
git init. Then, add all of the files to the project:
git add --all. Finally, make the first commit:
git commit -a -m 'initial commit'. The rest of the example commands in this post work from the idea that are you working in your project folder from the command line.
My git writing workflow uses two branches: “master” and “drafting.” As you might guess, drafting is a workspace, while master is the “best” version of the document or project. Using a second branch is completely optional, but I use it because it a) gives me piece of mind to know I’m not messing around on the master, and b) because it allows for a really granular level of revision and review for the times when you need or want it. Here’s my process:
- At the start of a session, I switch to the drafting branch if I’m not in there already:
git checkout drafting. If you need to create that branch, it’s
git checkout -b drafting.
- Now the hard part: write.
- When I’m done writing, I like to compare the changes I’ve made against the original and decide what to keep. I have a tendency to overwork a section, complicating it and/or making things worse. With the command
git diff drafting..master, I can see everything I changed in a writing session and compare it to the “master” version. If I use the same command with
difftool, it will open those changes in your editor. There are some expensive difftools like Kaleidoscope, but ol’ reliable TextWrangler also works just fine.
I’m diffing this blog post right now!
- Once I have my revisions sorted out, I commit the changes, switch back to my master branch, and merge the changes over.
git commit -a -m 'new prose'
git checkout master
git merge drafting
That’s all there is to it. Well, that’s actually the surface of what there is to it, but hopefully it’s enough to get you started. I would love to hear any more tips about how to use version control for writing workflows in the comments, or on Twitter.
13 Apr 2016
As I mentioned in part one of this series, one of the joys of Markdown is that it is a computer language, a kind of code. And as code, it can be interpreted, manipulated, and modified by a variety of programs and tools that make some of the more tedious parts of formatting automatic: clean and correct HTML tags, generating citations and bibliographies, converting to Microsoft Word, etc.
This post is about taking simple, easy to read Markdown and transforming it into something else. Because it focuses so heavily on footnotes and citations, I imagine it will be mostly of interest to other academics, but if you’ve done any sort of writing for the web, you might also find it useful.
Fair warning: each of these steps and programs I’m going to discuss has a much finer level of detail than I go into here. My goal is not to be exhaustive, but to explain the basic framework I use — both to finally have some personal documention, and to hopefully save one or two people from even a fraction of the Google searches I’ve done.
Making sense of footnotes and citations in Markdown can be tricky, especially since there are at least two styles of footnotes in widespread use:
- Inline footnotes, which look like this:
[^This is a full footnote.]
Reference footnotes, which use a unique identifier that points to a note somewhere else in your document:
This sentence needs a footnote.[^1] So does this one.[^website]
[^1]: Here it is.
[^website]: See [http://verifyandrepair.com](http://verifyandrepair.com)
If you are doing basic writing for the web, or writing something that might only have one or two notes, then either of these methods are fine. I used to be a nut for reference footnotes, but now I’ve come to enjoy inline. Switching between the two — even in the same document — isn’t a problem, since most Markdown parsers now recognize both.
Speaking of parsers, if there is one must-have tool in your Markdown toolkit, it’s Marked 2. Marked is ostensibly a preview window for Markdown files, but it’s also much more. It easily converts Markdown to HTML, but it also checks your spelling and grammar, verifies your links are working, exports to a variety of formats, and, as we’ll see below, previews your footnotes. It’s one of the handful of programs that I have open all day.
Sublime Text on the left, Marked 2 on the right. Click to enlarge.
Advanced Citation Using Zotero and Pandoc
If you’re working on a long document, planning on citing many sources, or using a bibliography, I cannot recommend a citation manager strongly enough. I feel like I’ve tried them all, and I keep coming back to Zotero. Sure its aesthetic is a bit dated, but it’s open source, free, and has extensions that make it a powerhouse little database.
One of those extensions is Better BibTex, which, when paired with the open source tool pandoc, allows you to fully automate formatting your citations and bibliographies. All you have to do is insert a citation key into a pandoc-style footnote — yes, this is a third footnoting style — which places the caret on the outside, like so:
^[@aitken76]. When you go to process that file, your blockquote…
…will magically have a nice footnote and bibliography entry.
For anyone who hates formatting footnotes and bibliographies, this really feels like magic. And while setting this up requires a bit of legwork up front, the payoff is worth it.
First, install Zotero, pandoc, and Better BibTeX (BBT) if you haven’t already. BBT will automatically create a “Citation Key” for every item in your Zotero library—you can even set the format for this key in the preferences. While you’re in the preferences, make sure to export the bibliography file you want to keep updated (I use my entire collection), and then set BBT to do an automatic export:
This creates a master bibliography file and keeps it updated.
Now you have two things: a set of master citation keys in Zotero and a master bibliography file. When the time comes to export your final draft to Word, PDF, or whatever, you can use Pandoc to parse your files, find the references, and insert them into the finished document.
Rocking the Pandoc
Pandoc is an incredibly powerful little command-line utility, but it’s just that: a command line utility. That makes it a little tricky, but as you’ll see, you can also use it in Marked 2 without ever going to the terminal.
The nice thing about command line utilities is that they follow a formula. In the case of pandoc, you give it a file to read and tell it what you want it to output. So, at the most basic level, you can run this command:
pandoc test1.md -o test1.docx, which takes your Markdown file and makes it into a Word document.
For our situation, though, we’re going to need to add three variables to that basic formula: one that tells pandoc to look for citations, one that points it to the bibliography file, and one that points it to a citation style.
- To have pandoc look for citations, you include a filter:
- To add the bibliograpy:
--bibliography=/path/to/bibliography (replacing that with the actual path to your file, of course!)
- Citation styles languages (CSLs) are the files that set style parameters, and they’re based on common guides: Chicago, APA, etc. There’s a CSL for pretty much every style out there, and you can find them in this repo. Download one, save it somewhere, and point to it just like the bibliography:
Putting this all together, you get this:
--filter pandoc-citeproc --bibliography=/path/to/bibliography.bib --csl=/path/to/style-file.csl
Our super basic conversion from before would now look like this in the terminal:
pandoc --filter pandoc-citeproc /path/to/file.md -o /path/to/output.docx --bibliography=/path/to/bibliography.bib --csl=/path/to/chicago-note-bibliography.csl
That’s a long script, but keep in mind that you can drag and drop files into the terminal, saving you the headache of typing out those paths. And since as many as three of those outputs might stay consistent (output file, bib file, csl file), then you could also save the string somewhere and just copy it in when you need it.
You can also use this system within Marked 2, and never have to touch the command line. You can add pandoc to Marked as a “preprocessor” which Marked will use to parse the file before it uses its own Markdown processor.
To do that, go to the “Advanced” tab of Marked’s settings. There, enable the custom preprocessor, and add the full path to your pandoc installation (hint: type
which pandoc in the terminal), and then add our string from up above. That will give you the full citation information in your previewed documents — which, again, you can export directly from Marked without ever having to go the command line.
The preferences in Marked 2.
At times, pandoc can feel both extremely simple and hopelessly complex, and there are many variations of settings (include export templates) that I just didn’t cover here. Please feel free to comment if you have questions, are stuck, or want to share how you do this in a more elegant manner.
Up next, in the final installment in this series, I’ll talk (hopefully more briefly!) about using
git for file management and version control.
11 Apr 2016
It’s been almost three years since I wrote a substantial post about my everyday writing workflow, with the exception of the occasional toss-off Markdown trick. But in those three years, even as my jobs have been less writing intensive, I’ve managed to improve my workflow, or rather, I’ve managed to do some things less incorrectly.
So I’ve cooked up a series of posts to share my current setup for everyday writing and blogging as well as for my academic book project. The good news is, it’s all the same workflow. Besides some small details, I use the same software and systems every time I sit down to type.
Here is a general outline of where we’re going:
- Part 1 (this post)
- Why Markdown?
- Creating a Markdown writing environment in Sublime Text
- Part 2
- Managing citations with Zotero
- Previewing documents in Marked 2
- Exporting documents via Pandoc
- Part 3
- File management and version control with Git
If you’ve gotten this far, you probably already have a passing knowledge and interest in Markdown, and/or have read one of several good tutorials on the subject. I’m not going to rehash them all here.
There are great, practical reasons to use Markdown: it’s plain text, human-readable, and can easily be translated into other formats. But for me, one of the most important concepts behind Markdown is that writing and formatting are two different things. This allows the endpoint of the process — whether that be a blog post, wiki entry, digital edition, printed syllabus, journal submission, book manuscript — to be separate from the work of drafting it. Microsoft Word uses the paradigm of the word processor: printed words on a sheet of paper. Markdown treats writing as code: iterative and malleable. Your mileage may vary, but for me, that notion has alleviated some of the pressure I feel during first drafts, or even sitting down to write in the first place.
Creating a Markdown Environment in Sublime Text
There are a variety of Markdown-informed and Markdown-flavored full-featured writing applications out there, including Ulysses and Scrivener. I’ve tried them all, and in my experience getting the most out of Markdown means using it “raw,” in files and folders as opposed to in databases.
With that in mind, when I sit down to write, I write in Sublime Text. It was designed for code, built to be fast, and is insanely customizable. That customization certainly comes with a learning curve, but after much trial and error, here is a glimpse of where I ended up:
A draft of this post in “solarized” Markdown with Critic Markup and a word count.
When you first download and run Sublime, it looks like a little black box. To change its visual appearance, you have to familiarize yourself with how Sublime handles customization, which is slightly different than other Mac apps. There are some settings that are available via the menus, but many others that are only available through XML files stored in the application’s library folder (~/Library/Application Support/Sublime Text 3/).
Say you want to change the color scheme. You can find a variety of options in “Preferences -> Color Scheme”. No code required! But if you want to permanently change the font size, you select “Preferences -> Settings - User,” which will open a file. There, you can add the line
"font_size": 17 to make things bigger. To learn which settings are where and what some of the defaults are, take a look at “Preferences -> Settings - Default.” That’s a master file that cannot be edited. To make any changes, copy and paste the lines into your User file. Experiment!
Also, note that you can view a sidebar of open files. This can also be entire folders of files, and you can save these sidebar arrangements as “Projects” from the menu bar. This allows you to create a draft or file management system like you see in apps like Scrivener.
You can add more robust customization to Sublime through community built “packages,” which work like browser extensions: they add extra functionality on top of the platform. To make working with them easier, install Package Control by following the instructions at that website.
Once you have Package Control installed, you can invoke it and add packages. I highly recommend starting with two: Word Count and MarkdownEditing. The first one does exactly what it says, and the second one adds some really handy extra Markdown features, including autocompletion of Markdown syntax like asterisks and brackets.
For more detail, this post does a nice job walking you through the process.
For even more flavor, I like to use Critic Markup, which allows you to make basic proofing marks and include inline comments. It’s not available as a package, so you have to manually install it to
~/Library/Application Support/Sublime Text 3/Packages. (Directions here.)
Once it’s installed, you can open a Markdown file and view it in Critic from the menu path “View -> Syntax -> Critic Markup -> Critic.” (You can also use this menu to open all Markdown files using Critic.) Critic works using a Sublime’s “syntax specific” settings, a feature that allows you to style your Markdown files differently from your plain text ones, your Python scripts, etc. You can see how this works in by looking at Preferences -> Settings -> Syntax Specific - User. There you can tinker with the font, hide the line numbers, and more. Here’s a link to my prefs for your reference.
When you look at these settings, you’ll see that they override your color scheme choice with one of their own. This makes sense — for the package to work, it creates a “language” (so it knows what strings to highlight) as well as a “theme” (the colors to make those highlights). These files are stored in the package folder, and, like anything in Sublime, you can mess with them. Since I wanted to keep my solarized look, I modified the color scheme and certain aspects of the language. I won’t cover that here, but feel free to have a look at my files while you tinker with your own: here’s the language, and here’s the theme.
Up next: citations, previews, and export.
26 Feb 2016
After spending some time with Reporter’s JSON output, I’ve been thinking about ways to leverage it as a input method for other tools and services. Perhaps the most obvious is Foursquare, since Reporter already pulls from the Foursquare API for its location data.
Getting this to work just requires a little API mojo and grep parsing of your Reporter questions.
- Make sure your Reporter app is saving your entries as JSON files to your Dropbox. This is in Settings ->Export Settings ->”Save to Dropbox.”
- Get a Foursquare API token. You can do this by registering an app with them. It doesn’t have to be an actual app, of course—you can just fill out the form to get your token.
- Make sure that your report contains these two questions: “What are you doing?” and “Where are you?” It can contain more than those questions, but it should at least contain those two.
- Make sure Hazel is watching your Reporter folder, which is in ~/Dropbox/Apps/Reporter-App. I use my favorite rule: “Date last modified” is after “Date last matched.”
Once you have that all set up, here is the script that Hazel should fire when your files change. You’ll need to add your filepath on line 18 and your API token on line 19. If you want your checkins to be public, you can either remove line 74 or change the value to “public.”
Then you should be off to the races. I’ve only tested this out for a few days, so if it’s crazy broken let me know and I’ll see if I can tweak it. This is my first time really working with
net/http — usually I take a shortcut and just use
curl — but I wanted to try making it “pure” ruby. Much of the credit goes to Paw for helping with that last API call.