Saturday, July 27, 2019

Statistics

"Best by test", said former-worldchampion Fischer about 1.e4. He got the best results with 1.e4 so he kept playing the same lines which brought him fortune earlier. Chess is about results so logically we like playing openings which have won us games before.

On the other hand some amateurs ditch an opening from the moment they have lost a game with it. They change openings at the same rate as they change their underpants. They play any opening but know nothing. They don't want to make any effort to find a solution for a problem in the opening or worse they think wrongly that the defeat is because of the chosen opening.

In the book Ivan's Chess Journey Unravelled the strong Dutch grandmaster Ivan Sokolov gives therefore the advise to test at least an opening a couple of times before taking the decision to give it up. Next to that he also made the interesting comment that he sometimes removes openings from his repertoire although they are theoretically sound. If the resulting positions don't fit your style and results remain under par then obviously you can't keep on playing them as a professional.

So statistics influence our choices in chess but in comparison with computerchess this is child's play. In computerchess statistics have always been used very intensively to make progress. Each new minor-upgrade of an engine is tested extensively not only to remove any bugs but especially to define any change of the playing-strength. Besides in the course of the history of computerchess we see an increasing use of techniques embracing statistics.

Speed had always been the key. The faster we can evaluate, the faster we can make new changes. The first logical step was to make automatic testsessions to avoid the slowness of the human operators. In the last decade as hardware kept gaining speed, it became also more and more interesting to choose for rather short and more than slow and long (games). This change of analysis I already explained in 2015 here see computers achieve autonomy and nowadays I use it myself regularly.

In the last 2 years programmers even started to work with only bullet-games or faster. The path was cleared for the neural networks which more than ever need huge amounts of games to learn. This way AlphaZero played against itself 44 million games and learned to play the best ever chess performed in history in just a couple of hours. It was only given the rules of chess and the rest was figured out by the engine. At least that is what we were told everywhere as you could read e.g. at chessbase. The real story behind was much more difficult to discover. To learn more about it I decided to buy the very hyped book Game Changer.
I am halfway with the book but meanwhile I do realize the book wasn't meant to explain the code of Alphazero. The authors are in the first place players so not surprisingly they concentrate on the chess-content of the games. It is still a nice book to read but to learn about the technical aspects of the engine, you better study the scientific document of Google/Deepmind.

That file explains that the engine works with a probability-distribution of 4672 possible moves. Knowing the starting position only contains 20 possible moves then this is clearly something very complex which not everybody can create. Even the experienced Leela-developers which used the input from Alpha Zero had still many questions. Maybe this was also the purpose of Deepmind. They try to give a new method to the world for creating a superstrong engine but leave things open so developers are forced to use their imagination.

Besides such very sophisticated learning-process is totally useless for a chessplayer. Playing 44 million games in a couple of hours is no option. The only statistics based on games played by humans with some value, are openingbooks extracted from a database (see my article green moves). Still even then the value is rather limited which I experienced first hand recently. Last season I played a game in which I played after the moves 1.e4 e5 with black 19 consecutive moves considered to be the most popular one at master-level. The Chigorin-variation of the Spanish remains today the undisputed number 1.
Only at move 21 I deviate from the 4 last master-games in the big-database 2019. It already had little statistical value but more remarkable is that black was landed in an unpleasant position. This combined with a well prepared and excellently playing opponent made a deadly combination.
[Event "Interclub KOSK - Deurne"] [Site "?"] [Date "2019.??.??"] [Round "?"] [White "Decrop, B."] [Black "Brabo"] [Result "1-0"] [ECO "C99"] [WhiteElo "2188"] [BlackElo "2309"] [PlyCount "71"] [CurrentPosition "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"] 1. e4 e5 2. Nf3 {(In our game of 2015 Benjamin chose for Nc3 but Nf3 was no surprise.)} 2... Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O Be7 6. Re1 b5 7. Bb3 d6 8. c3 O-O 9. h3 Na5 {(I have found in the big database 2019 still 1 game played in 2012 from Benjamin with this position. However his opponent played Bb7.)} 10. Bc2 c5 11. d4 Qc7 12. Nbd2 cxd4 13. cxd4 Nc6 14. Nb3 a5 15. Be3 a4 16. Nbd2 Bd7 17. Rc1 Qb7 18. Nf1 Rfe8?! {(I play this mainline already for 15 years but this is probably the last time.)} (18... Rfc8!? 19. Ng3!? Ne8 20. b3!? g6! 21. Ne2 Bd8!) (18... Bd8!? 19. Ng3 h6! 20. dxe5 dxe5 21. Qe2 Bc7! 22. Nh4 Bb6) 19. Ng3 Bd8 20. d5 {(In the book "the Chigorin Bible" Qe2 is given as the refutation but it is not so simple as some very recent correspondence-games demonstrate. I've not yet encountered before d5 in an official game but I had covered it in my personal notes as a critical test.)} 20... Nb4!? {(Almost anybody plays Nb4 but Ne7 is not really inferior.)} (20... Ne7!? 21. Nh4 Rf8!? 22. Bd3!? Kh8!? 23. Qf3! Bb6!? 24. Bg5! Nfg8 25. Rc2! f6!? {(Or first b4 and only then f6.)} 26. Bd2 b4 27. Rec1! g6!? 28. Qe2! b3 29. axb3 axb3 30. Rc4 Ra2 31. Rb1 Qa7 32. Rc3) 21. Bb1 Be7!? {(After the game my teammate Daniel Sadkowski criticized this move. I don't think this is really fair as it is very important to connect the rooks in this type of positions and other bishop-moves are unavailable. In my personal notes I found Na6 but I couldn't remember that in the game as I hadn't checked this line in my preparation.)} (21... Na6!? 22. Qe2! g6!? 23. Nd2! h5 24. Nf3! Ba5!? 25. Nh4!) 22. Nh4 g6 23. Bg5!? {(First Qd2 and next Bg5 is probably a little more accurate.)} 23... Qb6?! {(This idea is known and playable after 21.. .Na6 22.Nh4 g6 23.Bg5 Qb6 but here it doesn't work. The engine recommends the accurate Rec8 but white still keeps the better chances.)} 24. Qd2 Na6 25. Nhf5 {(After the game Benjamin told me that he had seen those sacrifices in the preparation. If you know that very strong players like Tkachiev, Bellaiche, Van Kampen and Bok have missed this in their games then it once again proves I am too predictable with my openings.)} 25... Bd8?! {(Gxf5 is no fun but the problems are become bigger now.)} (25... gxf5! 26. Bxf6 f4! (26... Bxf6?! 27. Nh5 Qd8 28. Qh6 Bh8 29. exf5 e4 30. Rxe4 Rxe4 31. Bxe4 Qh4 32. Rc3 Nc5 33. Rg3+ +- {[%eval 937,30]}) 27. Bxe7 Rxe7 28. Nh5 Qd8 29. Qe2) 26. Nh6+ Kg7 27. Rc3 Ng8 {(I saw white's next sacrifice after releasing my piece. In the postmortem I improved my play with the stronger Rf8 which makes the win for white a lot more difficult.)} 28. Nh5+ gxh5 29. Rg3 Bg4 30. Nf5+ Kh8 31. hxg4 h4 32. Rh3 Ra7 33. Rxh4 Bxg5 {(Also this time I only see the rooksacrifice after I released my piece. The position is beyond repair anyway but I should solve more tactical puzzles.)} 34. Rxh7+ Kxh7 35. Qxg5 f6 36. Qh5+ 1-0
Benjamin is a redoubtable opponent about which I wrote before already on my blog see ambitions part 2. Likely we can consider this game one of his better or even his best. For me this defeat was a sign to study properly again the Spanish Chigorin. 20 years of experience doesn't mean you are done with an opening. Fortunately I was lucky as last year a book about this opening was written by 2 grandmasters: Ivan Sokolov and the Spanish grandmaster Ivan Salgado Lopez.
As the authors notice in the book, it is weird no similar book was ever written before about this opening despite it is the most popular line of the Spanish. In the meantime I finished the book and I can definitely recommend it for anybody interested in this opening. We get a very good and extensive overview of the opening and it seems the opening is still fully playable. Also many improvements are given upon old evaluations. At chesspub the critic was given that it is incomplete. It is not totally unjustified as also 20.d5 isn't treated in the book. Still we shouldn't be too harsh either. With Na5 you can find more than 8000 mastergames in the big database 2019 and we still need to add the correspondence-games. In the book there are numerous alternatives given so you can get with ease around any missing lines with both colors.

Brabo

Monday, July 8, 2019

Computers achieve autonomy part 3

A non-chessplayer can't understand why somebody would love to stare for hours to a board with some wooden pieces while in the meantime the weather is nice to organize a barbecue with some friends. I rarely talk about chess with non-players. However also inside our chess-world there exists a lot of variety. The people playing competitions is the largest and most important group but we also have niches of which the computer-freaks are likely the greatest weirdos. They are the nerds of the nerds or some sort of super-nerd.

At least that was the case till recently as last year I noticed a clear change. The arrival of AlphaZero end of  2017 created a chain-reaction. This is very different from what we experienced after Deep Blue defeated in 1997 the reigning world-champion Garry Kasparov. At that time the revolutionary engine got dismantled leaving many questions unanswered. This time the momentum was kept as the code of the new engine was shared so other developers could create an AlphaZero for a standard PC. I am talking of course about lc0 or also called leela (more about this in a later article).

After decades of traditional alpha-beta programming we now see the steep and very spectacular raise of neural networks of which we could witness some very interesting clashes of styles in the most recent tcec-championships. Attractive games combined with a good marketing-strategy caused an explosive increase of chess-fans wanting to follow computer-chess see below graphic which presents the number of viewsessions per month for TCEC.
Bron: http://www.chessdom.com/tcec-season-15-to-ab-or-nnot-to-ab-that-is-the-question/








Getting 2 million viewsessions per month by just showing 1 game at once is definitely something extraordinary. Ok Carlsen does still much better but 99,9% of the other professionals never gets this kind of views. Besides this is not the only thing which proves that computer-chess is booming. On youtube we see a huge increase of videos in which games of engines are discussed. However even more stunning is that reporters now start to refer to games played by engines when they cover tournaments played by grandmasters. A couple of years ago this would've been totally unacceptable. The only reason of engine-games were till recently to understand which engine is stronger and almost nobody would value them equal to games played between humans. Well today some reporters do compare games played by top-grandmasters to a game played between Lc0 and Stockfish as happened e.g. in fide grand prix moscow semifinal chess.
[Event "FIDE Grand Prix Moscow 2019"] [Site "Moscow"] [Date "2019.05.24"] [Round "10.1"] [White "Grischuk, Alexander"] [Black "Nakamura, Hikaru"] [Result "1-0"] [ECO "E06"] [WhiteElo "2772"] [BlackElo "2761"] [PlyCount "107"] [EventDate "2019.??.??"] [CurrentPosition "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"] 1. d4 Nf6 2. c4 e6 3. Nf3 d5 4. g3 Be7 5. Bg2 O-O 6. O-O dxc4 7. Qc2 b5 8. a4 b4 9. Nfd2 c6 10. Nxc4 Qxd4 11. Rd1 Qc5 12. Be3 Qh5 13. Nbd2 Ng4 14. Nf3 Nxe3 15. Nxe3 a5 (15... a6 {(This game was started only a couple of hours earlier. It is very strange that the same opening occurs on the board while no openingbook was used nor any human input influenced the game.)} 16. Nc4 a5 17. Nfe5 Ra7 18. Rac1 c5 19. Qe4 Ra6 20. Nd2 Rd8 21. Nb3 Rxd1+ 22. Rxd1 f6 23. Nd3 Nd7 24. Rc1 Qe8 25. Nbxc5 f5 26. Qf3 Nxc5 27. Nxc5 Ra7 28. Qf4 Qd8 29. Nd3 Bd7 30. Qc4 g6 31. Nf4 Qb6 32. Bc6 Kf7 33. Bxd7 Rxd7 34. Nd3 Bd6 35. Qc8 Rc7 36. Qh8 Rxc1+ 37. Nxc1 h5 38. Qh7+ Kf6 39. Qh8+ Kf7 40. Qh7+ Kf6 41. Nd3 Qd8 42. b3 Be5 43. Qh6 Bc7 44. h4 Kf7 45. Qc1 Bd6 46. Qc6 Ke7 47. Qc1 Qh8 48. Qg5+ Qf6 49. Qc1 Qc3 50. Qg5+ Qf6 51. Qh6 e5 52. Nb2 f4 53. Nc4 Bc7 54. Qh7+ Qf7 55. Qh6 Qf6 56. Kg2 Qc6+ 57. f3 Qf6 58. Qh7+ Qf7 59. Qh8 Qf8 60. Qh7+ Qf7 61. Qh6 fxg3 62. e3 Qf6 63. Kxg3 Kf7 64. Kg2 Qg7 65. Qg5 Qf6 66. Qg3 Qe7 67. e4 Qd8 68. Qf2 Qd4 69. Qxd4 exd4 70. Nb2 Kf6 71. Kh3 g5 72. Nd3 Kg6 73. Nc5 Bb6 74. Nd3 Bc7 75. Nc1 Kf6 76. Nd3 Bb8 77. Nc5 Kg6 78. Nd3 Kf6 79. Nc5 {(Lc0 - Stockfish TCEC superfinal 15, round 78: 0,5-0,5)}) 16. Nd4 Ba6 17. Rac1 Rc8 18. Bf3 Qg6 19. Be4 Qh5 20. Bf3 Qg6 21. Be4 Qh5 22. Kg2 Ra7 23. h4 g6 24. f4 Qh6 25. Nb3 Kh8 26. Bd3 Bb7 27. Nc4 c5+ 28. Be4 Ba6 29. Nbxa5 Qf8 30. Bf3 Rd8 31. h5 Bf6 32. Rxd8 Qxd8 33. Rd1 Rd7 34. Rxd7 Nxd7 35. h6 Nb6 36. Ne5 Bxe5 37. Nc6 Nc4 38. Nxd8 Ne3+ 39. Kf2 Nxc2 40. Nxf7+ Kg8 41. Nxe5 c4 42. Bg4 Nd4 43. Ke1 Kf8 44. Kd1 Ke7 45. e3 Nb3 46. Nc6+ Kf6 47. Nxb4 Bb7 48. Be2 Na5 49. Kd2 Nb3+ 50. Kc3 Nc5 51. a5 Ne4+ 52. Kxc4 Nxg3 53. Bd3 g5 54. fxg5+ 1-0
Although the grandmasters started their game a couple of hours later than the engines, I do think this was just a coincidence. Nakamura had played this line already several times even just a couple of weeks ago in a rapid and such things aren't missed by a top-professional like Grischuk in his preparation. On the other hand I find it very remarkable that the engines manage to select a hyper-modern opening without using any openingbook nor any human intervention happened.

The fact that the engines can autonomously create games with interesting openings is something very important. Obviously top-players have discovered this too. Since 2010 I already maintain a database of games played by engines to use for my opening-analysis see e.g. using databases. I guess at that time I was an exception but today I am sure any ambitious professional does this even Magnus Carlsen. Well more likely his entourage takes care of it which I deduct from a twitter of his helper, the Danish strong grandmaster Peter Heine Nielsen: "Computer-chess is spectacular."

Besides those engine-games have also influenced Magnus' game. Many of the so called novelties already popped up in games played by engines. We see proof of this in the worldchampionship played against Caruana end of last year as in later tournaments.
[Event "?"] [Site "Shamkir AZE"] [Date "2019.04.08"] [Round "8"] [White "Sergey Karjakin"] [Black "Magnus Carlsen"] [Result "0-1"] [ECO "A00"] [PlyCount "78"] [EventDate "2019.03.31"] [CurrentPosition "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"] 1. e4 c5 2. Nf3 Nc6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 e5 6. Ndb5 d6 7. Nd5 Nxd5 8. exd5 Ne7 9. c4 Ng6 10. Qa4 Bd7 11. Qb4 Bf5 12. Qa4 Bd7 13. Qb4 Bf5 14. h4 h5 {(Magnus introduced this novelty already last year in the 12th game of the worldchampionship against Fabiano Caruana. However h5 was already known from the TCEC superfinal 11 played earlier in 2018 between Houdini and Stockfish.)} 15. Bg5 Qb8 16. Be2 a6 17. Nc3 Qc7 (17... Nf4 18. Bf1 Be7 19. Bxe7 Kxe7 20. g3 Ng6 21. Qb6 f6 22. Rc1 Nf8 23. Qe3 Qa7 24. Qd2 g6 25. Bg2 Nd7 26. O-O Qc5 27. Rfe1 Qxc4 28. Ne4 Qd4 29. Qxd4 exd4 30. Nd2+ Kf8 31. Be4 Re8 32. Kg2 b5 33. Bxf5 Rxe1 34. Rxe1 gxf5 35. Nf3 d3 36. Nd4 f4 37. Ne6+ Kf7 38. Nxf4 Ne5 39. Rc1 Re8 40. f3 Nc4 41. Nxd3 Re2+ 42. Kf1 Rd2 43. Rc3 Nxb2 44. Nxb2 Rxb2 45. Ra3 Rb1+ 46. Ke2 Rg1 47. Rxa6 Rxg3 48. Rxd6 Rh3 49. a3 Rxh4 50. Rb6 Ke8 51. Rxb5 Ke7 52. Rb7+ Kd6 53. Rb6+ Kc5 54. Rxf6 Kxd5 55. Rg6 Rh1 56. Kd3 h4 57. Rg5+ Ke6 58. Ke4 h3 59. Rh5 h2 60. Kf4 Ra1 61. Rxh2 Rxa3 {(Houdini - Stockfish TCEC superfinal 11 game 36: 0,5-0,5)}) 18. g3 Be7 19. Be3 e4 20. O-O O-O 21. Bxh5 Ne5 22. Be2 Qd7 23. Qa4 Qc8 24. c5 dxc5 25. Nxe4 c4 26. Nc3 b5 27. Qd1 b4 28. Na4 Be4 29. Qd4 Qf5 30. f4 Qg6 31. Bf2 Nd3 32. h5 Qf5 33. Bg4 Qxg4 34. Qxe4 Bd6 35. Qg2 Rae8 36. Bd4 Qxh5 37. Qf3 Qg6 38. Kh1 Re4 39. Bf2 Rfe8 0-1
The opening is the most obvious phase to learn from but also later phases can be instructive. The American top-grandmaster Sam Shankland would've never resigned below game if he had seen the 35th round of the tcec super-final of season 12. I already used the position in my article fake news to illustrate the gain of strength in endgames of Stockfish.
[Event "Tata Steel Masters"] [Site "Wijk aan Zee NED"] [Date "2019.01.25"] [Round "11"] [White "Anish Giri"] [Black "Samuel Shankland"] [Result "1-0"] [ECO "C50"] [PlyCount "89"] [EventDate "2019.01.12"] [CurrentPosition "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"] 1. e4 e5 2. Nf3 Nc6 3. Bc4 Bc5 4. d3 Nf6 5. Nc3 d6 6. Na4 Qe7 7. a3 Be6 8. Bxe6 Qxe6 9. b4 Bb6 10. O-O O-O 11. c4 Nd7 12. Bb2 Nd4 13. Nd2 f5 14. Rc1 c5 15. exf5 Nxf5 16. Ne4 cxb4 17. Nxb6 axb6 18. axb4 Qg6 19. Ra1 Rxa1 20. Bxa1 Nf6 21. Nxf6+ Rxf6 22. Qf3 Nh4 23. Qd5+ Kh8 24. f4 h6 25. Qe4 Qxe4 26. dxe4 Rxf4 27. Rxf4 exf4 28. e5 dxe5 29. Bxe5 Ng6 30. Bc7 b5 31. cxb5 Kg8 32. Kf2 Kf7 33. Kf3 Nh4+ 34. Kxf4 Nxg2+ 35. Ke5 Ke7 36. Bd6+ Ke8 37. Ke6 g5 38. Kf6 Kd7 39. Bf8 Nh4 40. Bxh6 Nf3 41. h3 Ng1 42. Bxg5 Nxh3 43. Be3 Kd6 44. Kf5 Kd5 45. b6 {(In this drawn-position the American top-grandmaster resigned. Many of this fans were shocked. If Samuel would've watched the TCEC superfinal 12 then he would've remembered that the same endgame was drawn easily by Stockfish in round 35 against Komodo.)} 1-0
Earlier an engine was nothing more than a tool. Today more and more people consider engines as an entity with its own identity.  Many players cheer for their favorite engine via the chat-box or in fora. Some even make their own website for them as you can see in this example mytcecexperience.

So we see less and less difference between games played by engines or humans. Last Tuesday Ex-Machina was shown once again at the tv but this seems for chess to be today more reality than fiction.

Brabo