Chess engine championship TCEC season 9, stage 3 terminated on first week of September 2016, and we can not argue without surprise. Stockfish had a powerfull lead, and Houdini, already 3 times winner of this championship, succeeded in qualifying for the final. One looser, Komodo, current champion, faced strong opposition, apparently stronger thant supposed to be qualified.

TCEC: Houdini won a place in final, is it better than Komodo?

Before entering Houdini and Komodo, performance analysis, it is worth knowing that I am neither supporting one or the other, neither working for any of them. My goal is to provide and share here some of my analysis and materials to let you forge your own opinion.

Just a context recall, TCEC season 9, stage 3 involved 8 chess engines, each of them had to face 8 times each opponent, thus leading to 224 showdowns beetween chess engines. During this stage, chess engines were granted rights to use same opening books up to a predefined plies depth, and also right to use table positions. You may read full rules documentation through TCEC hyperlink given above.

Which performs the best ?

Day after day, while watching some chess games played during stage 3, it was clear that a kind of classification will appear amidst chess engine performance. If true, this should be visible from a simple stat table. Let's have a look to stage 3, chess engine performance comparison table,

Three groups of chess engines seems to emerge

1. a group able to win more than 25% of their games, thus composed of Stockfish, Houdini and Komodo
2. a group able to draw more than 80% of their games, here only Fire is in the group
3. a group unable to avoid loss less than 25%. Subgroups could be consider here, by a split on 5% win, thus leading to a first subgroup with Andscacs, Jonny and Gull, and a second subgroup with Rybka.

As Houdini and Komodo share the exact same draw performance, ability to win makes the difference. At this game, Houdini has been slightly smarter than Komodo.

Does the playing side make any difference in performance ?

To ease analysis per engine, per side and per opening, I build following diagrams to sense performance difference. Use your Internet browser capabilities to increase image size or display them full size.

When playing light side, both Houdini and Komodo either win or draw. On 28 games, Houdini won 2 games more13 vs 11 than Komodo. Houdini performed better than Komodo when playing light side during stage 3.

When playing dark side, both engines recorded wins, draws and losses. On 28 games, Houdini won 1 game less than Komodo 4 vs 5, lost 1 game less 2 vs 3. Although different, results prove that engines reached same performance level when playing dark side.

Any lesson's learnt ?

Looking at openings played by Houdini with black, it appears, that its 2 losses where under Sicilian game 56% of points loss far more than current loss registered by this engine during stage 3. Therefore, Houdini opponents, should consider playing Sicilian opening willingly when playing light side.

For Komodo, 2 losses with black under Sicilian 40% of points taken and 1 under Catalan opening only 25% points taken. So, Komodo opponents should consider entering Catalan opening when playing light side. That's their best opportunity to score.

Note

Images made using many R packages.