I just ran across a paper from 2010 with the provocative (and Googleable) title "The Impact of a Peer-Learning Agent Based on Pair Programming in a Programming Course." (IEEE Transactions on Education 53(2) May 2010)
The premise is that beginning programmers interact with an intelligent tutoring system that alternates between a "driver" and "navigator/observer" role in pair programming. The overall workflow is that the programming task starts with a flowchart, and then code is written to implement what the flowchart shows. When the bot is in "driver" mode, it "asks for help" from the student (who has the role of navigator/observer) in figuring out the flowchart, by showing the student an incomplete flowchart and having her fill in the blanks. The bot then writes code for the flowchart, but makes deliberate mistakes (syntactic and semantic errors) and expects the student to correct them, so the student "learns by teaching".
When the bot is in "navigator/observer" mode, it observes student-entered code and responds to various common errors, both from bug libraries and from instructor-supplied "keyword-triggered" messages corresponding to common mistakes or commonly-chosen suboptimal problem-solving strategies (so there's no AI in this part of it).
(The bot's "navigator mode" was this group's previous work; what's new here is the bot alternating between the driver role and navigator role, even though the bot's implementation of those roles doesn't quite match pair programming—"pair programming inspired" might be more accurate).
In both modes, simple Bayesian models are used to estimate the likelihood of students making a particular error, based on concept-dependence graphs and observed mastery of the student on predecessor nodes in the concept graph.
A limited randomized-controlled between-subjects trial, in which students used the system bracketed by pre- and post-testing, showed that students exhibited better transfer (p>.05) and retained material longer (p>.05) when using the system that switched roles periodically than when using the system that was always in "navigator mode". The effect size was large (0.79) for transfer and small (0.38) for retention.
The student modeling isn't particularly sophisticated, and the "navigator mode" bot isn't a particularly impressive use of NLP or AI, but the idea of having a tutoring system switch between roles is intriguing. Conversation bots and program analysis have both made lots of progress since this paper was published; the possibility of doing a real pair-programming chatbot for scaffolded programming assignments is quite intriguing!
The premise is that beginning programmers interact with an intelligent tutoring system that alternates between a "driver" and "navigator/observer" role in pair programming. The overall workflow is that the programming task starts with a flowchart, and then code is written to implement what the flowchart shows. When the bot is in "driver" mode, it "asks for help" from the student (who has the role of navigator/observer) in figuring out the flowchart, by showing the student an incomplete flowchart and having her fill in the blanks. The bot then writes code for the flowchart, but makes deliberate mistakes (syntactic and semantic errors) and expects the student to correct them, so the student "learns by teaching".
When the bot is in "navigator/observer" mode, it observes student-entered code and responds to various common errors, both from bug libraries and from instructor-supplied "keyword-triggered" messages corresponding to common mistakes or commonly-chosen suboptimal problem-solving strategies (so there's no AI in this part of it).
(The bot's "navigator mode" was this group's previous work; what's new here is the bot alternating between the driver role and navigator role, even though the bot's implementation of those roles doesn't quite match pair programming—"pair programming inspired" might be more accurate).
In both modes, simple Bayesian models are used to estimate the likelihood of students making a particular error, based on concept-dependence graphs and observed mastery of the student on predecessor nodes in the concept graph.
A limited randomized-controlled between-subjects trial, in which students used the system bracketed by pre- and post-testing, showed that students exhibited better transfer (p>.05) and retained material longer (p>.05) when using the system that switched roles periodically than when using the system that was always in "navigator mode". The effect size was large (0.79) for transfer and small (0.38) for retention.
The student modeling isn't particularly sophisticated, and the "navigator mode" bot isn't a particularly impressive use of NLP or AI, but the idea of having a tutoring system switch between roles is intriguing. Conversation bots and program analysis have both made lots of progress since this paper was published; the possibility of doing a real pair-programming chatbot for scaffolded programming assignments is quite intriguing!
I've been thinking about just such a possibility for a long time. I wrote out a characteristic pairing interaction in RSpec a few years back: https://github.com/PairProgramming/RSpecExercise and keep on thinking about how bot pairing might be great for getting people prepared for real pairing.
ReplyDeleteThe thing I think about is using Ruby MetaProgramming to have the bot "work out" what needs to be coded to make a test pass - to do the simplest thing ...