Pair Programming MOOC Style - is it Agile?

This summer the “Engineering Software as a Service” (ESaaS) MOOC got re-branded as “Agile Developing using Ruby on Rails” (ADuRoR). This was a recommendation from edX and I felt it was probably a good move, since in the past there always seemed to be a few people surprised to find that the course was not about cloud computing per se, but about software engineering principles applied to web application development. The course was basically the same, although we were upgrading fully to Rails 4 and RSpec 3 and that was a fair amount of work in itself - more so in terms of updating materials than upgrading software. The big change we did make was to have pair programming become “required” for the first time, and have it assessed by peer review.

We’d introduced “optional” pairing some two years earlier. In the first instance, each assignment had a bonus 10% for submitting a video of pair programming on the assignment, but we also accepted solo videos or text summaries of the assignment talking about the pros and cons of pairing versus solo work. Part of our motivation for requiring more pair programming from the learners is our plan for a “part 3” or “project” capstone course. This will complete the ADuRoR Series, and will involve teams of MOOC learners collaborating on building software solutions for charities and non-profits around the world. We’re keen to ensure that this capstone course has teams that work effectively together. We strongly believe that pair programming is a critical part of an effective software development team. It’s not that you can’t develop great software without pair programming. It’s that you can’t work effectively in a team without good people and communication skills.

Somebody who would pair program for all the ADuRoR MOOC assignments would likely be better equipped to work effectively in a project team than someone who did not. Not that this is a guarantee; there are no guarantees, but we need some mechanism to ensure a high chance of success for our initial project teams. There’s also lots of evidence that pair programming has a big positive impact on learning to code.

As it happens the Agile process itself does not mandate using pair programming, test driven development, scrums, or anything of the sort. Let’s just refresh our memory as to the principles behind the Agile Manifesto:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Business people and developers must work together daily throughout the project.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Continuous attention to technical excellence and good design enhances agility.

Simplicity–the art of maximizing the amount of work not done–is essential.

The best architectures, requirements, and designs emerge from self-organizing teams.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

There’s so much here, but as is clear, tests and pairing are not mentioned explicitly, nor is Scrum or Kanban, or many other things that are commonly associated with Agile. Personally I see the last “stanza” as the critical one. Regular reflection on how to become more effective, tuning and adjusting behaviour accordingly.

This seems like extraordinarily good advice to me. I happen to really enjoy pair programming. Not everyone does, and it’s not right for every team; but the crucial thing is that in any period you look back and say well we tried X and it’s gone well, gone badly, made no difference. And then we try a variant of X, throw out X, continue X as appropriate.

Our hope with the project course is to have teams of learners operate in agile teams. Ultimately we want to empower those teams to evolve their own Agile process based on what works for them. However we’d very much like all those team members to have at least tried TDD, BDD, pairing etc. to inform their own Agile Process. We also believe that remote pair programming and remote scrum meetings will be a critical part of distributed project teams working effectively together. To the extent that you accept Google Hangouts (or similar video conferencing software) as a form of face to face conversation, then we can see this also fits in with the principles behind the Agile Manifesto.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

One of the main challenges for pairing in our MOOC has been the logistics of helping online learners find pair partners. While we had over 500 successful pairing sessions during the MOOC, some students have struggled to find pair partners available at convenient times for them, particularly on the later assignments since from start to finish the number of learners attempting the assignment goes down by an order of magnitude. It’s also clear that the logistical overhead of arranging a pairing session is not great for those on tight schedules.

Our options going forward appear to include:

1. reducing the amount of credit given to pairing; so it doesn’t feel as “mandatory”
2. improving the mechanism by which pair partners are found
3. improving the support and documentation of how to manage pairing

The danger with the first step is that it would likely reduce the number of people attempting to pair, and further exacerbate the problem of finding pair partners. We certainly don’t want learners to become frustrated with attempting to pair program; we do believe that both pairing, and reviewing other learners pair videos are valuable learning experiences. The ultimate question is can we reduce the pairing logistics overhead sufficiently through options 2 and 3 …

Are students/teams following the Agile process? How would you know?

I teach a large project course at Berkeley. Each semester, ~40 teams of 6 students each work on open-ended SaaS projects with real external customers. The projects complement an aggressive syllabus of learning Agile/XP using Rails; most of that syllabus is also available in the form of free MOOCs on edX and is complemented by our inexpensive and award-winning textbook [shameless plug].

We require the teams to use iteration-based Agile/XP to develop the projects. During each of four iterations, the team will meet with their customer, use lo-fi mockups to reach agreement on what to do next, use Pivotal Tracker to enter the user stories they'll work on, use Cucumber to turn these into acceptance tests, use RSpec to develop the actual code via TDD, use Travis CI and CodeClimate to keep constant tabs on their code coverage and code quality, and do continuous deployment to Heroku. Everything is managed via GitHub—we encourage teams to use branch-per-feature and to use pull requests as code reviews. Most teams communicate using Slack; we advise teams to do daily or almost-daily 5-minute standups, in addition to longer "all hands" meetings, and to try pair programming and if possible promiscuous pairing. Some teams also use Cloud9 virtual machines to make the development environment consistent across all team members (we provide a setup script that installs the baseline tools needed to do the course homeworks and project). There are also two meetings between the team and their "coach" (TA) during each iteration; coaches help clean up user stories to make them SMART, and advise teams on design and implementation problems.

While the non-project parts of the course benefit from extensive autograding, it's much harder to scale project supervision. Even with a staff of teaching assistants, each TA must keep track of 6 to 8 separate project groups. Project teams are graded according to many criteria; some are qualitative and based on the substance of the meetings with coaches (there is a structured rubric that coaches follow, which we describe in the "Student Projects" chapter of the Instructor's Guide that accompanies our textbook), but others are based on quantitative measurements of whether students/teams are in fact following the Agile process.

In our CS education research we've become quite interested in how to provide automation to help instructors monitor these quantitative indicators, so that we can spot dysfunctional teams early and intervene.

We've observed various ways in which teams don't function properly:

• Lack of communication: Teams don't communicate frequently enough (e.g., they don't do standups) or communicate in a fractured way, for example, a team of 6 really becomes two teams of three who don't talk to each other. The result is "merges from hell", lack of clarity on who is doing what, etc.
• Not using project-management/project-planning tools effectively: One goal of Pivotal Tracker (or similar tools such as Trello or GitHub Issues) is to plan, track, prioritize, and assign work, so that it's easy to see who's working on what, how long they've been working on it, and what has already been delivered or pushed to production. Some teams drift away from using it, and rely on either ad-hoc meeting notes or "to-do lists" in Google Docs. The result is poor ability to estimate when things will be done and finger-pointing when a gating task isn't delivered and holds up other tasks.
• Not following TDD: Projects are required to have good test coverage (85% minimum, combined between unit/functional and acceptance tests), and we preach TDD. But some students never give TDD a fair shake, and end up writing the tests after the code. The result is bad tests that cover poorly-factored code in order to meet the coverage requirement.
• Poor code quality: We require all projects to report CodeClimate scores, and this score is part of the evaluation. Although we repeatedly tell students that TDD is likely to lead them to higher-quality code, another side-effect of not following TDD is poorly-factored code that manifests as a low CodeClimate score, with students then manually fixing up the code (and the tests) to improve those scores.

There are other dysfunctions, but the above are the main ones. 

To help flag these, I've become interested in how to study not only teams and team dynamics, but what analytics can tell us about the extent to which students are following the Agile process (or any process). We recently summarized and discussed some papers in our reading group that address this. (You can flip through the summary slides of our reading group discussion, but it may help to read the summaries and/or papers first.)

One of these papers, Metrics in Agile Project Courses, considers a variety of metrics obtainable from tool APIs to evaluate how effectively students are following Scrum, especially by measuring:

• Average life of a pull request, from open to merge. “Optimal” is <24 hours “based on our past experience with the course”, but too-short lifetimes may be bad because they indicate the code wasn't reviewed thoroughly before merging.
• Percent of merge requests with at least one comment or other activity (eg associated task). This is similar to our metric of pull-request activity in ProjectScope.
• Mean time between a CI (continuous integration) failure and first successful build on the main development branch.
• All indicators are tracked week-to-week (or sprint-to-sprint or iteration-to-iteration) so instructors see trends as well as snapshots.
• Number of deploys this iteration (In their course, “deployment” == “customer downloads latest app”, but for SaaS we could just look at Heroku deploy history.)

In a more detailed paper (How surveys, tutors, and software help to assess Scrum adoption in a classroom software engineering project), the same group used a combination of student self-surveys, TA observation of team behaviors, and a tool they built called "ScrumLint" that which analyzes GitHub commits and issues relative to 10 “conformance metrics” developed for their course, including (eg) “percentage of commits in the last 30 minutes before the sprint-end deadline”. Each conformance metric has a “rating function” that computes a score between 0 (practice wasn't followed at all) and 100 (practice was followed very well); each team can see their scores on all metrics.

What's interesting is that they actually point to an Operations Research-flavored paper on a methodology for detecting nonconformance with a process (see the "Zazworka et al." reference in the summary for more details). The methodology helps you define a specific step or subprocess within a workflow, and rigorously define what it means to "violate" it.

With AgileVentures, we are trying to encapsulate some of this in a set of gems called ProjectScope that will be the basis of a SaaS app for Agile coaches trying to monitor multiple projects. Check back here periodically (or join AV!) if you are interested in helping out!

Pair programming with a bot partner?

I just ran across a paper from 2010 with the provocative (and Googleable) title "The Impact of a Peer-Learning Agent Based on Pair Programming in a Programming Course." (IEEE Transactions on Education 53(2) May 2010)

The premise is that beginning programmers interact with an intelligent tutoring system that alternates between a "driver" and "navigator/observer" role in pair programming. The overall workflow is that the programming task starts with a flowchart, and then code is written to implement what the flowchart shows. When the bot is in "driver" mode, it "asks for help" from the student (who has the role of navigator/observer) in figuring out the flowchart, by showing the student an incomplete flowchart and having her fill in the blanks. The bot then writes code for the flowchart, but makes deliberate mistakes (syntactic and semantic errors) and expects the student to correct them, so the student "learns by teaching".

When the bot is in "navigator/observer" mode, it observes student-entered code and responds to various common errors, both from bug libraries and from instructor-supplied "keyword-triggered" messages corresponding to common mistakes or commonly-chosen suboptimal problem-solving strategies (so there's no AI in this part of it).

(The bot's "navigator mode" was this group's previous work; what's new here is the bot alternating between the driver role and navigator role, even though the bot's implementation of those roles doesn't quite match pair programming—"pair programming inspired" might be more accurate).

In both modes, simple Bayesian models are used to estimate the likelihood of students making a particular error, based on concept-dependence graphs and observed mastery of the student on predecessor nodes in the concept graph.

A limited randomized-controlled between-subjects trial, in which students used the system bracketed by pre- and post-testing, showed that students exhibited better transfer (p>.05) and retained material longer (p>.05) when using the system that switched roles periodically than when using the system that was always in "navigator mode". The effect size was large (0.79) for transfer and small (0.38) for retention.

The student modeling isn't particularly sophisticated, and the "navigator mode" bot isn't a particularly impressive use of NLP or AI, but the idea of having a tutoring system switch between roles is intriguing. Conversation bots and program analysis have both made lots of progress since this paper was published; the possibility of doing a real pair-programming chatbot for scaffolded programming assignments is quite intriguing!

Pair Programming Considered Harmful?

One of the students in our distributed teams course posted to the forum there asking 

This course has been a fantastic introduction to distributed team management tools and techniques. I am curious, though, to read responses, defences or critiques, of pair programming. Are there effective ways to do distributed team management without it? I present this article as a starting point: not exactly a balanced and neutral viewpoint as its title makes clear, but one that may elicit response. 

http://techcrunch.com/2012/03/03/pair-programming-considered-harmful/

I was all ready to get out the big guns in defence of pair programming, but despite the provocative title I found the article reasonably balanced and a very interesting read.  Pair programming (like many other techniques) is no panacea.

You can read a summary of some more research on pair programming here:

https://en.wikipedia.org/wiki/Pair_programming#Studies

In my experience (and backed by various studies) pair programming works particularly well for learning developers, and for experienced developers when they are working on something new and complicated.

However I agree with the conclusion of the article author

The true answer is that there is no one answer; that what works best is a dynamic combination of solitary, pair, and group work, depending on the context, using your best judgement. Paired programming definitely has its place. (Betteridge’s Law strikes again!) In some cases that place may even be “much of most days.” But insisting on 100 percent pairing is mindless dogma, and like all mindless dogma, ultimately counterproductive.

You can do distributed team management without pairing, but without it you miss one important tool for helping your junior developers level up and distributing knowledge through the team.

Dogma is the danger.  Saying "never ever pair program" or "never ever work alone" is dangerous.  One needs to achieve a good balance.  The danger that I see for a lot of companies and organisations is that they reject pair programming out of hand.  They even have open plan offices, but everyone sits there with their headphones, working in their little silo, potentially working in a highly counter-productive fashion.

I would say follow the Agile process.  Every week you tinker and try things.  No pair programming this week?  Let's see what happens if we do 100% pair programming this week?  Everyone sick of pair programming - let's try all doing mob programming this week.  etc. etc.

The critical thing is to keep reflecting on what is (and isn't) working for your team.  Sometimes you need to be bloody minded to make a difference.  Other times you need to have an open mind.  Keep reflecting to work out when you need which :-)

If you can only speak TDD, maybe you'll think TDD?

I just finished reading Through the Language Glass, an interesting popular-press exposition of the relationship between spoken language and perception, and in particular of the "weak version" of the Sapir-Whorf hypothesis, according to which a speakers' use of language influences their mental processing.
An example is given of the Guugu Yimithirr aboriginal language, in which all location-specifying speech acts use only absolute compass directions (NSEW) as opposed to relative directions. For example, if we were standing next to each other, instead of saying "I am standing to the right of Armando", you'd say "I am standing to the north of Armando" (or whatever direction it happened to be). And if we then rotated clockwise together and faced a different direction, you'd then have to say "I'm standing to the east of Armando." G-Y speakers have to think constantly about their orientation relative to the world, since communication about place would be impossible otherwise. Experiments are described whose provocative finding is that this makes it more difficult, e.g., for G-Y speakers to solve spatial puzzles in which the relative spatial relationships between objects are unchanged but their absolute orientation is changed.
What does this have to do with Agile+SaaS? I've been looking for ways to more strongly encourage the students in the Berkeley course to really commit to TDD. (Many do, of course, but some still write tests "after the fact" only because they know we will be grading them on test coverage.) I wonder if one could devise a lab exercise in which students develop some functionality via pair programming, but they are only allowed to speak in terms of test results when discussing with each other. This would be an interactive (vs. autograded/solo) exercise, and the goal would be to get students to use a vocabulary that effectively limits their communication to describing things in terms of test results. Eg "For this next line, the effect I want is that given a valid URL, it should retrieve a valid XML document." I know many of us already (strive to) maintain this mindset, but is anyone up for designing/scaffolding an exercise that would help us teach it to TDD initiates?

Amazon Echo is a fun way to practice "API life skills"

Last week was my birthday, and I got an Amazon Echo. It's a remarkably cool device, but of course the coolest thing is you can write your own apps for it. (Technically they are "Alexa apps" and not "Echo apps"—Alexa is the back end that runs the apps, the Echo is a specific device that acts as a client, but there's also mobile client apps.) Amazon has a nice development environment in which you essentially write a simple "grammar" for the phrases you want your app to understand, and some "event handlers" that get called when particular phrases are recognized. You can write the event handlers in Node.js, Python, or Java. (No Ruby, but hey, I'm ecumenical and Python is just fine.)
Students of Part 2 of ESaaS know that I have strong feelings about the use of Node.js for server-side code. (This post does a great job of summarizing my views, as does this mildly NSFW xtranormal video.) Nonetheless, Amazon's developer documentation starter examples are mostly in Node, so I decided to give it a fair shake.

I decided to make an Alexa app that gives me upcoming departure information for BART, the Bay Area's rapid transit system, which I use daily to get to work. They have an RESTful XML API in which simple HTTP GETs return an XML data structure you can parse. OK, simple enough. All I would need to do in Node is do a simple GET of a URL, and find some XML parsing library so I can parse the result.

What does the code look like in Node.js to do a simple GET? It looks like this:


As a less-experienced JavaScript programmer, I had to stare at it for awhile to understand its flow, and then I realized my God, all it does is fetch a JSON data structure using an HTTP GET call. Does anyone else see what is wrong with this picture? What is app code doing trying to respond to individual HTTP-level events that occur at the transport level? This clearly violates the "A" in the SOFA principle I teach in ESaaS: a function should stick to a single level of abstraction. Plus the code is butt-ugly. I'm no more impressed with Node than I was at the beginning of the exercise.

Here's the code I wrote in Python to do the same thing (using BART's API):



But I digress. The point is writing Alexa apps is a great way to build up your API-fu, since the best Alexa apps will probably receive concise speech commands, interact with several back-end APIs, and present concise speech output. And you can write and test the apps without having a device, using Amazon's web interface to test how the app responds to specific phrases. Plus the apps can be pretty fun to use, and the device makes me feel like I live in a home of the vaguely near future.
So try out writing some Alexa apps—you can even deploy them free on Lambda, a "managed AWS" service in which you deploy handlers rather than full VMs (essentially).
But for the love of all that is good…write them in Python.

Looking under the hood in cloud datacenters

There's a great article in the May 2016 issue of Communications of the ACM authored by a group of Googlers, including former Berkeley PhD student David Oppenheimer (now the tech lead on Kubernetes) and my PhD advisor Eric Brewer (Berkeley professor on leave to be Google's VP Infrastructure), on the evolution of cloud-based management systems at Google. They were doing "containerization" of cloud servers before almost anyone else, and contributed to the Linux containerization code. The article distills over 10 years of designing and working with different approaches to containerizing software to share the hardware in Google's datacenters, and some lessons learned and still-open problems. A couple of key messages:

• The "unit of virtualization" is no longer the virtual machine but the application, since modern systems like Kubernetes and Docker virtualize the app container and the files and resources on which it depends, but actually hide some details of the underlying OS, instead relying on OS-independent APIs. Apps are therefore less sensitive to OS-specific APIs and easier to port across containers.
• As a result of having both resource containers and filesystem-virtualization containers map to an app (or small collection of apps that make up a service), troubleshooting and monitoring naturally have affinity to apps rather than (virtual) machines, simplifying devops.
• A "label" scheme in which a given instance of an app has one or more labels simplifies deployment and devops as well. For example, if one instance of a service is misbehaving, that instance can (eg) have the "production" label removed from it, so that monitoring/load balancing services will exclude it from their operations but leave it running so it can be debugged in situ.

The article also points to still-open problems, including (surprise!) configuration (every configuration DSL eventually becomes Turing-complete and they're all mutually incompatible), dependency management (related to configuration, but also introduces new complications, i.e. what if service X depends on Y but Y must first go through a registration or authentication step before it can be deployed), and more.

Overall a really interesting read that suggests, ironically, that the programmer-facing view of cloud datacenters is actually a "back to the 1990s" view of managing a collection of individual apps with APIs (remember ASPs?) rather than managing a collection of either virtual or physical machine images.