Watson vs Cal vs Stanford

[andreaborn]

Thanks to Todd Crain's generous offer on this board to get some of us into the audience, I got to see the Watson-Stanford-Cal demo match last night -- very interesting and so much fun!

This article does a good job describing the IBM presentation, basics of the match, and outcome, so I'll just add a bit of color. http://www.stanforddaily.com/2011/11/18 ... 9s-watson/

The crowd was overwhelmingly pro-Stanford (our venue), but after the match started with Watson's run of 2 categories, the entire auditorium erupted in cheers for Berkeley, the first human team to get in on the buzzer/signaling device. Cheers were louder for Stanford than Cal throughout, but it was mostly a "humans rooting for humans" evening -- primarily due to those first 10+ questions. There's been a lot of buzzer discussion in the media and on this board, and it continued in the Q&A and at the stage after the event ended last night. I was a not-reticent participant in that, but will take comments over to the Watson vs My Wife thread to avoid repetition here.

Watson was set to take categories in order last night rather than go DD-hunting and the humans did hunt, but Watson still found 2 of the 3. Cal successfully went TDD on a "70s sports teams" clue for 1800 (I think) -- TDD was necessary given the scores, but still pretty brave given the kids' ages and the fact that it was the first clue chosen in the column.

Last but not least, Todd was fabulous -- soooo funny before, during, and after the match -- keeping it lively and being both encouraging and snarky to all parties when appropriate, as when Watson repeated a wrong answer. Loved the interchange between Todd and Eric Brown from IBM about the development of the vulgarity filter (IBM name) / porn filter (Todd's characterization). And all this when he'd been up since 3 am NY time, so he'd been awake for ~20 hours on who-knows-how-much sleep.

I don't know what sort of project(s) Todd has in the works now that his Watson career is over, but I certainly hope he lands something deserving of his talents, charm, and generosity (he invited us down to the stage to chat before and after the match, personally took pictures of us with Watson and immediately sent them along, made sure we got Watson hats, etc.).

Todd, thanks so much again, and congratulations on a great show and performance.

Go Bears! (Big Game this weekend...)

[Rex Kramer]

. . . made sure we got Watson hats . . .

Watson hats??? I didn't get a Watson hat! What do you gotta do to get a Watson hat? Jeez!

Rex

[Volante]

Did Stanford's band storm the stage during Final Jeopardy before time ran out?

[andreaborn]

Did Stanford's band storm the stage during Final Jeopardy before time ran out?

No, but come to think of it, there was a tree laying on its side in the hallway just outside the auditorium....

[reddpen]

. . . made sure we got Watson hats . . .

Watson hats??? I didn't get a Watson hat! What do you gotta do to get a Watson hat? Jeez!

Rex

Blow a transistor?

[TheConfessor]

LICENSED MERCHANDISE FOR $2000:
==============================
THIS IBM HEADGEAR CAN BE
OBTAINED BY SENDING $15.99 TO
http://ibmlogogear.com/ProductList.aspx?did=14503

What do you gotta do to get a Watson hat?

[TheConfessor]

Last but not least, Todd was fabulous -- soooo funny before, during, and after the match -- keeping it lively and being both encouraging and snarky to all parties when appropriate, as when Watson repeated a wrong answer. Loved the interchange between Todd and Eric Brown from IBM about the development of the vulgarity filter (IBM name) / porn filter (Todd's characterization). And all this when he'd been up since 3 am NY time, so he'd been awake for ~20 hours on who-knows-how-much sleep.

I agree with everything you said about Todd. He respects the game and its traditions. He knows that the contestants are the stars of the game, not him, but he's not an automaton and he's very good at injecting a little humor where appropriate. Most of the practice matches I played at the IBM Watson Lab were hosted by Eric Brown, who impressed me with his abilities to run a smooth game, especially considering that he is a Ph.D. computer scientist and not a professional actor.

[ToddAlanCrain]

Thanks to Todd Crain's generous offer on this board to get some of us into the audience,

Last but not least, Todd was fabulous -- soooo funny before, during, and after the match -- keeping it lively and being both encouraging and snarky to all parties when appropriate, as when Watson repeated a wrong answer. Loved the interchange between Todd and Eric Brown from IBM about the development of the vulgarity filter (IBM name) / porn filter (Todd's characterization). And all this when he'd been up since 3 am NY time, so he'd been awake for ~20 hours on who-knows-how-much sleep.

I don't know what sort of project(s) Todd has in the works now that his Watson career is over, but I certainly hope he lands something deserving of his talents, charm, and generosity (he invited us down to the stage to chat before and after the match, personally took pictures of us with Watson and immediately sent them along, made sure we got Watson hats, etc.).

Todd, thanks so much again, and congratulations on a great show and performance.

I was THRILLED that you were able to make it to the game! And, I enjoyed our Buzzer Battle discussion afterwards as well. While we can't convince the other person our argument is the "right" one, at least we can respect one another to be civil about our disagreement.

The "Porn Filter" story is one of my favorites because it was the first time Harry Friedman was in attendance and I felt a tremendous amount of pressure to put on the best game(s--we played 6 every day) possible.

Because of my marathon 23 hour day, I brought a nice cold with me back to NY. Lovely.

Thanks so much for the kind words. I appreciate it very much!

I agree with everything you said about Todd. He respects the game and its traditions. He knows that the contestants are the stars of the game, not him, but he's not an automaton and he's very good at injecting a little humor where appropriate.

Thank you for saying that. This was never discussed at IBM, but I adopted, as my main job, bringing a sense of humanity to all of the games I hosted. Every player we had during the 2010 sparring matches was, in their own way, nervous about competing against Watson. No one had been exposed to what he was capable of and that brought an extreme amount of tension to the game...tension that I felt distracted from the fun of the whole experience. So, that's what I did. I created this "game show host character" that made this experience less threatening for our contestants, while still respecting the style and flow of the game...something I'm very proud of.

[heelsrule1988]

Blow a transistor?

[ToddAlanCrain]

Here is a link to one of the stories about the CAL vs. STANFORD vs. WATSON game:

http://sanfrancisco.cbslocal.com/video/ ... rcomputer/

[TheConfessor]

Todd, the video clip showed "Lock-In" and "Lock-Out" indicators for the three players. I haven't seen that before. Does "Lock-Out" mean that the contestant hit his buzzer too early? If so, that might be useful feedback for human contestants who are trying to improve their buzzer timing.

[seaborgium]

Thank you for saying that. This was never discussed at IBM, but I adopted, as my main job, bringing a sense of humanity to all of the games I hosted. Every player we had during the 2010 sparring matches was, in their own way, nervous about competing against Watson. No one had been exposed to what he was capable of and that brought an extreme amount of tension to the game...tension that I felt distracted from the fun of the whole experience. So, that's what I did. I created this "game show host character" that made this experience less threatening for our contestants, while still respecting the style and flow of the game...something I'm very proud of.

I wouldn't say I was nervous, just excited to compete in a very very accurate simulacrum of the game, against contestants I remembered from TV in years past. All the same, you did help make our experience less lab-ratty, and I hope our paths cross again somehow.

[ToddAlanCrain]

Todd, the video clip showed "Lock-In" and "Lock-Out" indicators for the three players. I haven't seen that before. Does "Lock-Out" mean that the contestant hit his buzzer too early? If so, that might be useful feedback for human contestants who are trying to improve their buzzer timing.

Yes. The game board system that IBM uses for it's "demo" (or press event) games is different than the one used on the broadcast version of the show. Clearly, IBM does not own the rights to the Jeopardy game board, so they had to create their own which includes this information. "Locked Out" for a team that has been beaten to the buzz. "Locked In" for the team that won the buzz. "Try Again" for teams whose buzz register at the same time...meaning that no one won the buzz because the system registered two teams buzz in the same instance. "Too Early" for obvious reasons.

The actual Jeopardy system does not give the feedback that this system does. This was primarily for the researchers use early on in the process--Watson's "infant" period. Before we were allowed to have the Jeopardy system to work with at the beginning of 2010, the researchers needed to create a system where they could read every nanosecond of the process after the buzzer indicator lights went off. They needed this to see how Watson did in competition with the other players buzzer skills. At the very beginning of the process, Watson was not as aggressive a player as he is now (clearly).

Since this system was created by IBM, it wouldn't help future players because it's not available for purchase. Plus, the actual Jeopardy system is not equipped to handle 6 individual buzzers divided up into two teams--the way we play some of our demo games.

I wouldn't say I was nervous, just excited to compete in a very very accurate simulacrum of the game, against contestants I remembered from TV in years past. All the same, you did help make our experience less lab-ratty, and I hope our paths cross again somehow.

Thanks so much! It makes me incredibly happy to know that I accomplished my mission.

[alietr]

Every player we had during the 2010 sparring matches was, in their own way, nervous about competing against Watson. No one had been exposed to what he was capable of and that brought an extreme amount of tension to the game...tension that I felt distracted from the fun of the whole experience.

I'm not sure I'm understanding this correctly ... but did you call Watson "he"?

[ToddAlanCrain]

I'm not sure I'm understanding this correctly ... but did you call Watson "he"?

I'm the only one that did...and does.

If I host the game like the contestants were playing against a computer, it takes the fun out of it. Especially for me. It's easier to make jokes about something that you can make "enjoyable" to everyone in the room instead of something that you ignore (like a chair or a table), except when he answers questions.

I've done a lot of green screen work, so playing with something that isn't in the room or isn't "alive" is something that I enjoy immensely. I've also worked with the characters on Sesame Street (I did a national tour with Sesame Street Live for 10 months). Same concept. You have to act as if the big yellow bird is an actual human being that you can interact with...keeping in mind it's limitations. My job in every situation similar to that was to make whatever I was playing against something "human."

Watson (he) is no different, in my mind.

[larkin1734]

This is terribly late, but I thought I'd add just a few things that might interest the board. Andrea has already given an extensive recap; here are a few other notes/observations.

Nico Martinez, 2005 College champ, was on the Stanford team. He's at Stanford Law now. I believe one of the Berkeley players was on the high school team 2x champ Chris Fleitas coaches (this wasn't mentioned at the match--it's merely from remembering the player at a different venue).

The exhibition board was pieced together from previous games. Ten of twelve categories were from the last two years; one was from 2004, and one was from 2006. FJ was from 2010.

With two exceptions, the clues were the same as on the show. Two clues were replaced because the originals were presented by the Clue Crew, and there were no audio or video clues at the exhibition. Why IBM didn't just use categories without video clues instead of changing individual clues, I can’t say.

There were five instances of categories appearing in different rounds. Four of the categories in the J! round of the exhibition appeared in the DJ! round when they aired, and one DJ! category was in the J! round originally.

Todd credited one answer that would have been negged on the show (an extra letter was added to a movie title). He acknowledged the answer was wrong but allowed it because it was the first time in about 25 games anyone had come close. By this point of the match, Watson had a huge lead, so it didn't affect the outcome.

Scores (End of J! round, End of DJ! round, Wager, Total):
Watson 17600 33300 499 33799
Berkeley 4200 16400 16399 32799
Stanford -400 8800 8800 17600

Coryat (59/60 answered correctly)
Watson 23400 (29 R including 2 DD, 2W)
Berkeley 11800 (15R including 1 DD, 2W)
Stanford 8800 (15R, 4W)

Todd was a great host and struck a balance between honoring the game that was responsible for the gig and taking care of the humans onstage, entertaining the audience all the while.

This was the third and final exhibition match involving colleges. I believe IBM is done with the exhibitions after the charity auction match happens (it may have taken place by now).

Thanks for everything, Todd!

[alietr]

Todd credited one answer that would have been negged on the show (an extra letter was added to a movie title). He acknowledged the answer was wrong but allowed it because it was the first time in about 25 games anyone had come close.

Any recollection of what the clue was? That's a low get rate ...

[larkin1734]

Todd credited one answer that would have been negged on the show (an extra letter was added to a movie title). He acknowledged the answer was wrong but allowed it because it was the first time in about 25 games anyone had come close.

Any recollection of what the clue was? That's a low get rate ...

It was the $1600 clue in Candied Cinema from this game: http://www.j-archive.com/showgame.php?game_id=3549. Stanford added an "S" at the end, rendering the answer plural.

To be fair, I have no idea who the participants were the previous times this clue was run. Eric Brown, the IBM guy who led the discussion portion of the event, did mention an example where a child gave an unexpected answer. Given the state of communication today, it's not entirely surprising that a student answered "Facebook friend invites" to the $1600 Lincoln Blogs clue from this game: http://www.j-archive.com/showgame.php?game_id=129. So I know that IBM has made presentations to kids, although there is no evidence to suggest what groups of people had seen this Candied Cinema clue before.

[Paucle]

Wow, really? Surprising, since that YA novel seems to be a school standard now. Very few of my nieces and nephews haven't had to read it for English class since it came out. Of course, most attend a parochial school, which is its setting.

[ToddAlanCrain]

Todd credited one answer that would have been negged on the show (an extra letter was added to a movie title). He acknowledged the answer was wrong but allowed it because it was the first time in about 25 games anyone had come close.

Any recollection of what the clue was? That's a low get rate ...

It was the $1600 clue in Candied Cinema from this game: http://www.j-archive.com/showgame.php?game_id=3549. Stanford added an "S" at the end, rendering the answer plural.

To be fair, I have no idea who the participants were the previous times this clue was run. So I know that IBM has made presentations to kids, although there is no evidence to suggest what groups of people had seen this Candied Cinema clue before.

To repeat what I said during the game at Stanford, NO ONE had EVER answered that question correctly during my time with the clue. Even when we played it during the original sparring matches at IBM in Yorktown Heights, NY. Stanford's answer was the closest ANYONE had ever come to getting it right. They gave me "The Chocolate WarS" when the correct answer was "The Chocolate War"...as a display of my generous nature (and, realizing that awarding their team the money wouldn't change the outcome of the game...but possibly allow them to stay in the game long enough to play Final Jeopardy!), I told them that since they were the only team to ever answer that clue with even a SLIGHTLY correct response, I gave them the money. This wasn't the "official rules" decision to make, but no one was walking out of that theatre with a check for their final winnings.

The point of the Watson demo games was to show the technology in a fun and entertaining way. This was Watson's "novelty" stage. He'd already proven himself, and this was a live example of how he works and plays the game...nothing more. Although, I expect my players to play to the best of their abilities, I explicitly explain to them that this game is to be as entertaining for the audience as possible and the FEEL of the game I run is the complete opposite from the way the game is presented on the broadcast version. I encourage all of the contestants during the demo games to say anything they'd like during game play itself. The funnier, the better. I take on the role as ring leader for the circus that is my version of the game. Different from how I ran the games while at IBM, but just as enjoyable.