Before entering Houdini and Komodo, performance analysis, it is worth knowing that I am neither supporting one or the other, neither working for any of them. My goal is to provide and share here some of my analysis and materials to let you forge your own opinion.
Just a context recall, TCEC season 9, stage 3 involved 8 chess engines, each of them had to face 8 times each opponent, thus leading to 224 showdowns beetween chess engines. During this stage, chess engines were granted rights to use same opening books up to a predefined plies depth, and also right to use table positions. You may read full rules documentation through TCEC hyperlink given above.
Which performs the best ?Day after day, while watching some chess games played during stage 3, it was clear that a kind of classification will appear amidst chess engine performance. If true, this should be visible from a simple stat table. Let's have a look to stage 3, chess engine performance comparison table,
- a group able to win more than 25% of their games, thus composed of Stockfish, Houdini and Komodo
- a group able to draw more than 80% of their games, here only Fire is in the group
- a group unable to avoid loss less than 25%. Subgroups could be consider here, by a split on 5% win, thus leading to a first subgroup with Andscacs, Jonny and Gull, and a second subgroup with Rybka.
Does the playing side make any difference in performance ?To ease analysis per engine, per side and per opening, I build following diagrams to sense performance difference. Use your Internet browser capabilities to increase image size or display them full size.
Any lesson's learnt ?Looking at openings played by Houdini with black, it appears, that its 2 losses where under Sicilian game 56% of points loss far more than current loss registered by this engine during stage 3. Therefore, Houdini opponents, should consider playing Sicilian opening willingly when playing light side.
For Komodo, 2 losses with black under Sicilian 40% of points taken and 1 under Catalan opening only 25% points taken. So, Komodo opponents should consider entering Catalan opening when playing light side. That's their best opportunity to score.
Images made using many R packages.