Evaluating Non-Traditional Digital Humanities Dissertations

Most of the online advice I’ve encountered for evaluating non-traditional scholarly projects seems to focus on seeking jobs or applying for tenure with a portfolio that is significantly digital; the process of translating that body of work for a reviewing committee matches up pretty well with how evaluation works for the dissertation defense process for a non-traditional and/or digital project (and I’ve covered some of these suggestions at the end of this post). Charting progress and recording effort throughout the non-traditional dissertation, though, is a mostly undiscussed topic.

Having a unique format scrubs a lot of the traditional methods for evaluating these factors: maybe there’s no “chapter” to turn in, or you’ve undertaken a critical building project in a programming language your committee can’t read (and even if your committee could read through the entire body of code you author, you wouldn’t want them to need to—just as with a monograph project, you want to be respectful of mentors’ time by asking for feedback at specific milestones and with some kind of concise artifact to assess).

My dissertation has a unique format—there’s no monograph, and the bulk of the work is blogging, design, coding, user-testing, and analysis of testing data. I’ve come up with a comfortable set of practices that let me keep my dissertation committee updated and on-board with my expected outcomes, give me some handy accountability to produce a reasonable amount of weekly work, and clearly document my effort and progress in case administrators even request evidence of the equivalency of my research with that of students using a more time-tested format. This post will cover how I’m recording my progress and sharing recognizable milestones with my advisor and committee;  I use a combination of GitHub, Zapier (an IFTT-like service), Basecamp, scheduled emails, and GoogleDocs.

GitHub for evaluating any type of writing

I’m using GitHub to manage my code (more below), but it turns out GitHub has a lot of nifty ways for me to let my advisor and committee have a sense of my coding progress and effort—visualizations that would work just as well for someone using GitHub to track changes in their writing, for example. I’ve added my advisor as a collaborator on my dissertation code repository so that he can also see the progress charted there.

List of commit messages for GitHub code

Example code commits give a summary of what the code I’m adding to the repository achieves (e.g. did it fix a specific bug? add a new feature?).

Whether or not the text you’re working with is code, GitHub lets you track frequency and quantity of work, and offers some nifty visualizations to give you an overview of your work habits. With a project where you’re learning or advancing a technical skill, this can be a nice way of showing that even if something you spent two weeks on didn’t pan out (e.g. I ended up not using my Islandora digital edition work), you were doing active work during that time.

GitHub visualization options include (note all examples are from a repo opened at the end of August):

  • Viewing the number of code commits made by week:

GitHub bar chart of code commits by week.

  • Evaluating your work habits by showing a “punchcard” of what days of the week and times of day you commit the most code (mostly sticking to my commitment to not dissertate on weekends so far!):


GitHub "punchcard" chart of code additions and deletions by day and time of day.

  • Charting code additions and deletions by week:

GitHub chart of code additions and deletions by week.

 GitHub for code tracking



I’m managing my dissertation code via a completely private repository that regularly gets pulled into a private repository that only certain collaborators can view (sharing read-only code to only select people is slightly roundabout).

I’m using GitHub to backup and track my coding work for all the regular webdev reasons (lots of copies keeps stuff safe, it will be easy to release code to the public when it’s ready, etc.). Part of my deliverable work is good documentation of my coding, so I need to keep the repository README updated. (A repository, aka “repo”, is the place on GitHub where my code is stored, which you can visit and view as webpages; a README is a text file sharing some information about that repo, visible at the bottom of the front page of a GitHub repository).

Screenshot of the README file from my code repository.

README file explaining what the code in the repository does/will do.

I’m using the README in two ways:

  1. A place to explain what my code does, why I made certain technical choices, and (once the repository becomes publicly accessible) how anyone can use my code for their own research (e.g. dependencies, installation and configuration instructions, where to ask for help).
  2. A place to track and visibly credit the various pieces of FOSS (free and open-source) code that my project builds off.
Screenshot of README file describing provenance of code in my repository.

Tracking the provenance of various code collections I’m building off.

My project focuses on creating and testing new interface abilities by building off existing wheels such as Drupal (for the site structure itself) and Annotator.js (for the core textual annotation functionality). Many of the terms of the various FOSS licenses on these code collections dictate that I attribute this work properly, but even without such licenses visible and detailed attribution would be extremely important to me. Everyone should get credit and thanks for sharing their work with others! (See the awesome “Collaborators’ Bill of Rights” that came out of a MITH workshop for more on why correct credit should matter to everyone). Plus, making it clear that this project is founded on a variety of existing code sets might encourage others to think about platforms that could emerge from combining or augmenting existing code, giving us new tools and approaches instead of sinking time into reinventing wheels. Since I’ll also be producing a lot of my own reusable code (plugins, interface), this page will also help me make it clear to future employers what pieces of code are authored by me.

Basecamp for project progress

Basecamp is a project management tool that also works well for teaching courses (and it’s free for teachers!) or working on solo projects. Among other features, it lets you create various task lists containing tasks with due dates:

Screenshot of tasks on a Basecamp task list

Some of the tasks in my Basecamp project; this list is helping me track what needs to be done before I start alpha-testing.

Basecamp will also help me automatically turn feedback from user-testing into tasks on my dissertation to-do list, using Zapier, a service that connects various other internet services (e.g. get an email when one of your tweets is RT’d, and other IFTTT-like options).

Screenshot of using Zapier to turn GitHub issues into Basecamp tasks

Using Zapier to create Basecamp tasks from GitHub issues.

GitHub repositories allow people to create “issues”, which are like comments on the code or software in the repository; I’ll then place labels on these issues depending on what type of problem they represent: a bug, a feature request, or something out of scope for the dissertation project. Once issues get a label, they automatically appear on a Basecamp task-list as to-dos containing the text in the issue and assigning me a due date to address that issue in the next two weeks. You can get started similarly managing GitHub issues using this Zap by using the template I’ve shared.

Screenshot of issues in my GitHub repository

Some example issues on my GitHub repository; each issue automatically became a to-do with a due date on my Basecamp project after I labeled the issue, thanks to Zapier.

Emails and GoogleDocs

I’ve scheduled myself to create a weekly email to my advisor covering three things:

  1. what I accomplished on the dissertation over the past week,
  2. what my goals were for the following week, and
  3. anything I needed him to do or read.

These emails are both useful in keeping my advisor up to date with my work and as a record of that progress, should we ever need to demonstrate it to university administration (although I’ve gotten written consent for my unique dissertation format from the interested parties as described in this post, so I’m not expecting to actually need these records). Weekly emails are also effective for me as a carrot and/or stick: I know I need to make some progress during the week or the email will be empty, and writing the email reminds me of all the things I got done in a week.

My four dissertation committee members have been awesomely willing to meet with me as a team. I’ve met with them all in the same room twice so far at milestones during my dissertation process (once while designing the dissertation and preparing the prospectus, and once when I was starting to have a good idea of what the final project deliverables would look like). We’ll be meeting again in the next weeks as I finish site functionality and begin user-testing.

Meeting as a team has been a great way to make feedback into a conversation among several areas of expertise. It’s also been great to reaffirm that the committee understands what the deliverables at the time of the defense will look like, and that they still find that outcome satisfies the requirements of a dissertation in my department. Once I find a meeting time that works for everyone, I send them an agenda for what I’ll cover in the meeting; this helps me use their time wisely, as well as let them know the type of feedback I’m seeking or any questions I might need them to answer (e.g. I let them know I’d be checking if they still felt my deliverables worked as a full dissertation, so they can revisit my project beforehand and come to the meeting prepared with any questions or concerns). To make getting prepared for the meeting easier, I’ll also share a GoogleDoc highlighting what I’ve accomplished since our last meeting; how my project, timeline, or expected deliverables have changed (if at all); and a list of linked titles for the blogs I’ve published since the last meeting.

Evaluation at the end of the dissertation

That’s how I’m tracking my work throughout the dissertation, but how do you establish when a non-traditional dissertation fulfills the normal checks on dissertation completion—has the project demonstrated the ability to design and pursue a scholarly research project at the doctoral level? If you parallel the dissertation with the tenure-application process, you increase the amount of available reading about digital humanities research evaluation. There are guidelines like the MLA’s for digital projects, the Center for Digital Research in the Humanities’ resource page on “Promotion & Tenure Criteria for Assessing Digital Research in the Humanities“, and the Carolina Digital Humanities Initiative’s resource page on “Valuing and Evaluating DH Practice”. Digital humanities scholars who seek tenure are publicly blogging about their experiences more often, and tips about packaging for tenure review blog posts, tweets, and other examples of scholarly public engagement are frequent among digital humanists on Twitter. Just as tenure applications are starting to include ways scholars have engaged the public, keeping track of online reactions to your project (blog post reading stats, Twitter comments, and RTs for tweets about your dissertation) can serve as a testament to the meaningfulness of your research and dissertation format, as well as offer a kind of shallow mid-stage peer review (thinking along the lines of: retweets of links to posts about your project may be an extremely mild and imprecise form of peer review).

Would you like to volunteer as an early user of the Infinite Ulysses site? Sign up here!

Infinite Ulysses Technical Update: Summer 2014

The core functionality of my website is the ability to highlight pieces of the Ulysses text and leave annotations on them (questions/answers, interpretations, contextualizations, notes); to achieve this, I’m building off Annotator.js—Javascript code that lets you do what I just described, with a variety of plugin code that lets you add neat features like a heatmap of where the most annotations appear on a text (e.g. on what pages did readers have the most questions? what pages had the most contentious interpretations?). Since the site itself is built with Drupal, I’m using the Drupal modules already built to let Annotator.js work with, for example, Drupal’s database as the place to store readers’ annotations.

Not as easy as it sounded (is it ever?)

That the Drupal log under the skull and crossbones.

That’s the Drupal logo under the skull and crossbones. This was on my home dry-erase board for a while…

My original project plan was to spend a lot of time coding the look and behavior of the reading interface, with the site functionality not re-inventing various wheels, but instead making connections among existing sets of code that allow textual annotation and social commentary, voting, ranking, etc. In practice, the existing Drupal-module versions of Annotator.js are buggy. Two big examples (for me):

  • the Drupal Annotation modules (Annotation and Annotator) don’t store tags added to annotations, so any tags added by a reader disappear forever on refreshing or changing the page (and tags really need to be treated as entities in their own right, and tied into the core Drupal taxonomy); and
  • the code that sets up an individual textual annotation to be an “entity” (a Drupal type of content—such as a blog post, book page, or comment—that other existing Drupal modules automatically know how to act on) isn’t completely working either (e.g. you can’t add fields—such as a field to add tags to an annotation, fixing the tag storage problem—with the current code).

Having everything not go as smoothly as planned was actually great, though! I had a realistic goal in terms of what I needed to get done by the end of the summer to stay on track (have the site’s main functions almost all working, ready to be completely finished at the end of September), and I wasn’t worried about doing different work than expected to achieve those goals. Having the Drupal annotation modules need more work than expected meant I got to spend the summer trying various things to solve these issues, and learning a lot in consequence—like creating non-basic custom Drupal entities.

For example, I learned how to make a custom entity fieldable (able to have fields, like a place where you can add tags and have them autocomplete if they match existing tags) and add some fields to it, both by writing code (“programmatically”) as well as via the Drupal admin interface (after making the annotation visible to the core Field UI module). I also read through the Annotator code so many times that I became much more comfortable with it, so I’ll be able to augment it in the future far more easily.

In the end, it looks like I’ll be building off the code fix created by another DH developer who was kind enough to start an online conversation with me about our mutual work improving textual annotation in Drupal (more on that project once they’ve decided to make their code public). I went back and forth about whether I wanted to build off their solution or keep hacking at my own, but in the end I hadn’t been expecting to need to make those code fixes in the first place, so I decided I’d keep my remaining coding time this year free for expected work with the actual reading interface, like making a heatmap of the annotations and porting other existing Javascript/Coffeescript plugins to work with the Drupal version of Annotator.

Other summer work

Screengrab of many many tabs open across the top of my browser.

Screengrab of many many tabs open across the top of my browser. A fairly normal view when I’m coding!

Patching. I submitted an extremely simple patch to the Drupal Annotation module (fixing a misspelled variable) as a way of learning the process; my patch was accepted into the module. Now I know how to create patches when I want to share further work with others using Drupal Annotation.

Installation profiles. I’m currently working on two separate local versions of the Infinite Ulysses website: one to work on improving the Annotation function code, and one for work on everything else (e.g. user profiles, site content, theming). As I work on the site’s annotation abilities, I fairly often want to reinstall the site as a slightly quicker alternative to weeding all the references to annotation code out of the database (I should set up uninstall hooks to do this for me eventually…), or to see whether the code works correctly when installing the modules into a new site. Keeping the annotation work on a separate site lets me do this without losing my other work on the site.

But! I don’t want to start from a standard Drupal blank site every time I need to reinstall my annotation code; I want a sample page of Ulysses to test the code on, user permissions already set up, modules installed, and various other things configured that I’d rather not do manually every time I reinstall the site. Installation profiles are a way to avoid starting from square one with a Drupal site, instead automatically pre-configuring the site as you’d like it to be (think of it like a module that tells the site to configure and install things a certain way during the site installation process). When I visit install.php, I now have a “Infinite Ulysses Pop-Up” option in addition to standard or minimal Drupal installations. Eventually, this profile will turn into a distribution, such that anyone can use it to install a Drupal site pre-configured to act as a participatory digital edition.

Annotation sidebar. The Annotator.js code I’m working with displays annotations as pop-ups hovering over the text, but from experience with my UlyssesUlysses prototype and usability research, I prefer having annotations authored and displayed in a sidebar. This will also be handy for the tablet version of the website, as you’ll be able to slide the annotation sidebar off and on the page like a drawer. I’ve been playing around with a couple ways of achieving this: modifying this existing CoffeeScript or this extension of that code (demo’d here) to work with Drupal, creating the sidebar by resetting the current CSS on annotation pop-ups and styling them into a sidebar, or a tactic such as this repo which purports to display annotations as Drupal comments.

Book-page styling. I’ve been vacillating a bit as to the final display structure of the novel: I’m pretty sure I either want to have one page per chapter of the book and infinite scroll to put all the text of a chapter on one page, or have separate pages to match up with each page of the 1922 text of the novel (which I’m using). This is something I might end up doing A/B testing on, showing the text to some visitors one way and to other visitors the other way, and comparing their reactions and stats such as time spent on the site. Here are some questions I’ll need whatever solution I use to address:

  • How many annotations can load on a page at a time? How does this limit or non-limit affect site speed?
  • How do annotations load on a very long page that only loads partial sections of the text at a time (when they’re scrolled to, i.e. infinite scroll)?
  • How will bookmarking on the site work—will a reader’s last visited and furthest page in the book be marked as where they should start next, or will users have the option (or necessity) to bookmark one or more pages themselves?
  • If I do offer one long scrolling webpage per chapter of the novel, can I easily generate bookmarks that jump a reader down to the last line they were reading?
  • What other ways can users chart their progress through the book, such as badges per chapter, or a progress bar for pages read?
  • What if readers want to read it out of order, or only some parts online (and mark parts they read offline as completed)? What if they want to glance ahead without losing the bookmark at where they actually want to pick up reading next?

Long-term thinking. I met with the artist and IP lawyer behind the excellent web/iPad comic of Ulysses (Ulysses Seen). We’re all keenly interested in opening Ulysses to more public reading and conversation; they want a way for comic readers to be able to jump from the comic to the text, and I’m also interested in supporting the comic’s readers in enjoying the text—so we’ve been discussing combining our websites into one super-accessible, textual and visual Ulysses edition, after I’ve defended my dissertation. This is also a good way for me to think about how I’ll manage hosting costs and moderation effort in the future.

Other summer activities included lots of reading (as much as I read, I’m always adding more links to my reading list every time I go on Twitter), researching the social modules I want to use on annotations, and drafting site content. In particular, I’ve been looking into how existing communities (especially online ones where people submit text and/or converse) have succeeded in building friendly spaces, self-moderating, creating community guidelines, and fostering an inclusive and welcoming atmosphere.

Would you like to volunteer as an early user of the Infinite Ulysses site? Sign up here!

User-testing a digital edition: Getting the feedback you need

I’ve been thinking about specific questions I want to ask during the user testing for Infinite Ulysses (my dissertation project)—more specifically, Rachel’s tweet had me thinking about how to describe to user-testing volunteers what kind of feedback I’m seeking.  I came up with some statements that model the types of thoughts users might have that I’d like to know about. For the first phase of beta-testing on my project, I’ll ask testers some abbreviated form of:

Have one of the following thoughts, or similar? Please elaborate:

  1. “I wanted to see ___ but didn’t / couldn’t locate it / stopped looking for it even though it’s probably there”
  2. “This is broken”
  3. “This is cool”
  4. “This could be done better (by ___?)”
  5. “This doesn’t work how I expected (and that is good / bad /should be changed ___ way)”
  6. “Where is ___ again (that I used before”
  7. “This requires too many steps / is hard to remember how to use”
  8. “Don’t see how / why I’d use this”
  9. “I’d use this ___ way in my reading / teaching / work”
  10. “I gave up (where, when)”
  11. “____ would make me like this site / like it better / use it regularly”
  12. “I’m not interested in Ulysses, but I’d like to use this interface for a different novel / non-fiction text / (other thing)”
  13. “Starting to read the text on the site took too long (too much to learn / too much text or instruction to wade through) / took the right amount of time (intro text and instruction was appreciated or easily skippable, or site use was intuitive enough to get started)
  14. “I would recommend this site (to x person or y type of person)”
  15. “The problem with this site is ___”
  16. “Reading on this site would be easier if ___”
  17. “I wish you’d add ___”

Testing stages for Infinite Ulysses

As I start to get all my design and code ducks in a row for this project this month, I’ll be moving into a cycle of user-testing and improving the website in response to user feedback. I’ll be testing in four chunks:

  1. Self-testing: Working through the site on my own to locate any really obvious issues; for example, I’ll step through the entire process of signing up and reading a chapter on the website to look for problems. I’ll step through the site with different user personas in mind (imitating the different backgrounds and needs of some of my target audiences, such as first-time readers of Ulysses and teachers using the site with a class). I’ll also apply various website assessment tools such as validators for code and accessibility.
  2. Alpha testing: Next, I’ll run some low-stakes testing by inviting my dissertation committee, close friends, and family to break the site. This might get me to a point where the next stage of testers aren’t hitting any problems big enough to take the site down or make testers wait while I take days to fix a difficult issue.
  3. Beta testing: I’ll conduct beta-testing this fall and spring by opening the site to exploration and use by the people who have generously volunteered via this sign-up form. Phase I will take place this fall and take feedback from volunteers using the site individually; Phase II will take place in winter and early spring, continuing individual use of the site, and adding in people using the site in groups, such as teachers with their classes, or book clubs reading together.
  4. Post-release testing: I’ll continue to take feedback once the site goes live for use by anyone in June 2015, although I’ll need to scale down work on requested enhancements and focus on bug fixes and continued data gathering/analysis on how people use the site to read. Setting up site logging and Google Analytics on my site will help me monitor use as time allows.

User testing how?

I’ll be building on my user-testing experience from my master’s research and the BitCurator project, as well as trying some new tactics.

The thesis for my information master’s degree involved a use study exploring how members of the public (and others with a content interest in a website, but lack of experience with digital humanities and edition/archives commonplaces) experienced scholar-focused DH sites, using the Blake Archive and Whitman Archive as examples. I was particularly interested in identifying small design and development changes that could be made to such sites to better welcome a public humanities audience. For my master’s research, I built off existing user study metrics from a related field (learning technology) as well as creating and testing questions suggested by my research questions; feedback was gathered using a web survey, which produced both quantitative and qualitative data for coding and statistical analysis.

I’m hoping to further set up

  • web surveys for willing site visitors to fill out after using the site
  • shorter web pop-up questions—only for users who check a box agreeing to these—that ask quick questions about current site use (perhaps incentivized with special digital site badges, or with real stickers if I can get some funding for printing)
  • in-person meetings with volunteers where I observe them interacting with the site, sometimes having them talk aloud to me, or with a partner, as to their reactions and questions as they use the site
  • various automated ways of studying site use, such as Google Analytics and Drupal site logging

For bug reports and feature requests, site visitors will be able to send me feedback (either via email or a web form) or submit an issue to the project’s GitHub repository. All bugs/enhancement feedback will become GitHub issues, but I don’t want to make users create a GitHub account and/or figure out how to submit issues if they don’t want to. I’ll be able to add a label to each issue (bug, enhancement request, duplicate of another request, out of scope for finishing my dissertation but a good idea for some day, and won’t fix for things I won’t address and/or can’t replicate). I’m using Zapier (a If This Then That -like service) to automate sending any issues labeled as bugs or enhancements that I want to fix before my dissertation defense to Basecamp, in an appropriate task list and with a “due in x days” deadline tacked on.

To read more about user testing in the digital humanities, check out my posts about

User testing for the long haul

I’ve got one major technical concern about this project (which I’ll discuss in a later post), and one big research-design concern—both related to the “Infinite”-ness of this digital edition. My research design concern is the length of this user-testing; I’m pursuing this project as my doctoral dissertation, and as such I’m hoping to defend the project and receive my degree in a timely manner. Usability testing can be done over the course of a few months of users visiting the site and my iterating the design and code; testing use and usefulness, as in

  1. how people want to use the site (i.e. perhaps differently from how I imagined),
  2. how people read Ulysses (a long and complex book which, if you’re not attempting it in a class or book club, might take you months to read), and
  3. what happens to a text like Ulysses as it accrues readers, their annotations, and the assessment the social modules lets readers place on others’ annotations (the more readers and annotations, the more we can learn)

are things I can begin to gather data on, and begin to speculate on what trends that data suggests we’ll see, but I won’t be able to give them the full treatment of years of data gathering within the scope of the dissertation. To address this, I’ll both analyze the data I do gather over the course of months of user testing, and try to automate further data-gathering on the site so that I can supplement that analysis every few months or years without requiring too much effort or funding to sustain this work.

Would you like to volunteer as an early user of the Infinite Ulysses site? Sign up here!