T-invariant is launching a series of materials on the state of the supercomputer industry in Russia and the world. Today we will tell you about the most prominent player in this small but strategically important market — the company T-Platforms and its founder — Vsevolod Opanasenko, who has been living under a criminal case for the last five years. T-invariant describes the story of the “Russian Jobs” of the Putin era and tells how a manufacturer of computing machines at the level of the top lines of the world TOP-500 gained fame in the international market, was able to get out of the US “black” list, but got burned on the supply of cheap computers for traffic police exams under a fabricated case. The Opanasenko case allows us to see what happens when security forces “harass” businesses, and in what ways people in uniform transfer leading engineers to work “behind the fence”.
Sergey Abramov, Corresponding Member of the Russian Academy of Sciences and the country’s leading supercomputer specialist, listened to Putin’s nineteenth address to the Federal Assembly in his home in the Yaroslavl region, recovering from a two-week “imprisonment” in the Yaroslavl Regional Psychiatric Hospital, while getting to know materials of his criminal case (more than 900 pages in total) and the conclusion of psychiatrists about his full sanity. During his speech, Putin set the task of increasing the total capacity of domestic supercomputers by 10 times by 2030.
Having heard this, mathematician Abramov went to the site TOP-500 world ranking of supercomputers, took the indicators and quickly calculated the potential progress of the total power of the best world machines by 2030. He could do this because by that time he was only under a written undertaking not to leave the country and had the opportunity to engage in scientific and analytical work. And just recently he spent many months under house arrest, he was prohibited from engaging in any intellectual activity because the Russian FSB suspects him of donating $75 to the Anti-Corruption Foundation of Alexei Navalny (the criminal case has been going on for a year and a half, and court hearings are currently underway).
At the Lebedev Prize ceremony of the Russian Academy of Sciences, 2015. S. Abramov’s Facebook. (From left to right: Sergey Abramov, Vladimir Fortov, President of the Russian Academy of Sciences (2013-2017), Vsevolod Opanasenko, owner of T-Platforms)
Abramov’s calculations led to simple results. According to his mathematical model, by 2030, the world’s best supercomputers will have to increase their capacity by at least 30 times. That is, if we assume that the goals set by Putin are achieved, by 2030 the best Russian computing machines will lag behind the world’s by three times.
Now all that’s left to do is figure out who and how will increase the total capacity of domestic supercomputers, given that two of the three most powerful Russian companies in the industry have been destroyed by the Ministry of Internal Affairs and the FSB, their founders have been under investigation for many years, and the remnants of the industry entered 2024 planning to celebrate a decade without new domestic machines in the world’s TOP-500.
Putin included the topic of supercomputers in the national project “Data Economy” and announced that he had allocated funds for them for his next six-year term700 billion rubles. It should be noted that by the time of his instruction, two years had already passed since all global manufacturers of chips, processors and other components had put a stop to all direct commercial relations with Russia and Russian companies. However, there remains gray and black imports (the euphemism “parallel” is used in Russian official vocabulary). In fact, if we talk about the Russian market of advanced supercomputing, then in the last 10-12 years two major players with their powerful engineering teams and interesting large projects have been working on it. These are RSC Group of Companies and T-Platforms.
T-invariant decided to figure out how computing technologies developed (supercomputing is, of course, one of the brightest markers of the development of any serious country) during the period of early Putinism and Medvedev’s “thaw” and what went wrong after 2012, when the security forces looked into these “clusters”. We will tell you why in some countries engineers and entrepreneurs are pushed towards two nanometers of processors (a nanometer is one ten-millionth of a meter) and two exaflops of supercomputer performance, while in others they are pushed mainly towards two years in a pretrial detention center. This is exactly what happened to the founder of T-Platforms, Vsevolod Opanasenko.
The main news about the life of scientists during the war, videos and infographics are in the Telegram channel T-invariant. Subscribe so you don’t miss it.
Supercomputers in the “national economy”
There is no formal definition of supercomputers. There are only descriptive characteristics of machines that can be classified as supercomputers. These are computers with the greatest computing power that devices have achieved to date. For example, ENIAC, built in the late 1940s and designed to calculate artillery tables, was certainly a supercomputer of its time. Comparing its performance with today’s most budget smartphone is simply ridiculous. The public rating of the TOP-500 most powerful machines today is constantly updated. All devices from this list are supercomputers by definition. Another criterion for a device to belong to the class of supercomputers is the price. Typically, the cost of supercomputers starts at $100 million.
The difficulty of defining a supercomputer is also due to the fact that today many typical supercomputer tasks are successfully solved on computing clusters of not the most powerful machines. If you look at the TOP-100 – the first hundred of the TOP-500 – you will not find a single machine from Google there. It’s not because Google doesn’t have money for a supercomputer, but because the company has a different computing strategy: Google works on clusters of machines.
The LUMI supercomputer is one of the fastest computer systems in the world. Photo: https://www.lumi-supercomputer.eu
The TOP-500 (especially the top part) looks a bit like the World Exhibition of “national economic achievements”. The largest “pavilion” is the USA, smaller “pavilions” are in Europe, Japan, Korea, China. Switzerland has a very impressive “pavilion”. Russia also has one, but much more modest. The prestige factor is important here. The TOP-500 mainly includes machines from government laboratories and organizations. The machine from the Oak Ridge National Laboratory of the US Department of Energy has been in first place for several years, the machine from the Argonne National Laboratory of the same ministry is in second place. The third place is occupied by a supercomputer of a private company (this is a Microsoft machine). In general, there are few supercomputers from private companies in the top hundred. Cluster technology is probably more flexible and adapts better to changing conditions of the consumer market. But Google’s radical position was not supported by Microsoft, Amazon, or Nvidia. Their machines are quite high in the rating. When the “startup” Nebius needed to make a name for itself, the company built a supercomputer, and it took 19th place in the rating. This is, among other things, also powerful advertising.
Russian machines are also in the top hundred – three supercomputers from Yandex (42, 69 and79 places) and one Sber machine (83 place). This looks rather modest. But it is necessary to take into account that not all supercomputers are included in the public rating. There are companies and government organizations that prefer not to openly announce their achievements and capabilities. Defense departments do this more often than others.
The supercomputer market today is relatively small. One of the estimates: 9.13 billion US dollars in 2023. The market is expected to grow at a CAGR of 9.4% from 2024 to 2030 and reach US$15-16 billion. (To put the entire segment into perspective, compare the supercomputer market to, for example, the laptop market, which was worth $170 billion in 2023, or the smartphone market, which was worth more than $600 billion.)
Supercomputer of the T-Platforms company. Photo: https://habr.com
The supercomputer market today (as always) is mainly formed by government orders, although it provides its capabilities to private companies. The main areas of use of supercomputers are energy, space, weather forecasts, logistics tasks, economic forecasts and other applications that require high computing power. In recent years, the development of artificial intelligence has been added to these areas, which many countries have declared a priority (namely countries, not private companies).
The most important part of the tasks that supercomputers solve is formed by defense orders. In fact, one of the main tasks of supercomputers is to ensure the sustainability of state infrastructure.
One of the main methods that supercomputers use to solve problems is computer modeling of complex processes. In an interview with the Scientific Russia portal on August 8, 2024, Corresponding Member of the Russian Academy of Sciences Igor Petrov listed some of the tasks of this type facing Russian science and the economy today. The first task the scientist named was seismic exploration, including in the Arctic. This is a state program that is mainly implemented by state corporations. Methods for solving this kind of problem were proposed back in the 18th century by Euler and Lagrange. They wrote out a system of partial differential equations, but analytical solutions to these equations are unknown and are unlikely to be found. Therefore, they began to be seriously solved only in the 20th century, when with the development of computers it became possible to do this approximately. Such approaches were also used to solve other equations that describe the explosion of a nuclear or thermonuclear bomb and, for example, the processes of flow around a spacecraft. Already in the 1960s, the most powerful computing equipment that existed at that time was used for these problems. In fact, these were the supercomputers of that time.
Another task that Petrov named is modeling a seismic wave during an earthquake, when powerful vibrations roll across the earth’s surface and destroy foundations. In order to build reliable houses that will not collapse like a house of cards, the effect of such a wave must be calculated. The very task of seismic exploration requires the creation of so-called digital twins of oil and gas-bearing regions. This is necessary, for example, when studying the natural reserves of Yamal. Among the tasks that need to be solved first, Petrov names ensuring the safety of rail transport (here the customer is Russian Railways). This is especially important for high-speed trains. Such a train creates a strong seismic wave that destroys embankments and damages the tracks. All this must be monitored and taken into account during design. This is also done using computer modeling methods and requires very large computing power.
Petrov does not mention this, but today, another approach has appeared to many of the tasks he listed – machine learning. AI models are being created that can solve problems that ten years ago were solved exclusively by methods of approximate calculations. AI models can give an approximation no worse, andoften better, but for high accuracy we need supercomputers.
At the end of the conversation, the correspondent asks Petrov: will it be possible to solve all these important and difficult problems? And the corresponding member answers: “With the help of supercomputers – yes. And we don’t have enough of them. We desperately need them: it’s a security issue. So we hope and wait.”
But will the corresponding member of the Russian Academy of Sciences Igor Petrov wait?
Between the four “i’s” and Crimea
The last time the Russian authorities showed a truly serious interest in supercomputing was at the beginning of Dmitry Medvedev’s presidency in 2008. This is the opinion of a former top manager of one of the state corporations, who spoke to T-invariant on condition of anonymity. “The commission on modernization was meeting then, remember it? Many smart people created high-quality state programs for technological development, they met in the newly built Skolkovo. And there was also the concept of the four “I”s. As far as I remember, that was the last time they made strategic-level documents where they even talked about supercomputers. And what can we talk about now? What computing tasks are there in Russia in 2024? What should be considered in the civilian market? There is no supercomputer industry anymore. And there are no documents, programs, strategies where this word is used,” our interlocutor is sure.
Awarding the Development Prize in the Best Project in Industry category. Photo: http://government.ru
“The last domestic public car that entered the world TOP-500 was assembled in 2014, ten years ago,” says Sergei Abramov. It was Lomonosov-2, which was made for Moscow State University by T-Platforms. The project took 22nd place in that year’s ranking. In 2015, T-Platforms delivered the JURECA system to the German supercomputer center Juelich (in that year’s ranking, it took 49th place in the world TOP-500). “But this was more likely inertia, 2014 was a turning point,” explains Abramov.
Most experts agree that the most fruitful years for the Russian industry fell between 2008 and 2015. “Yes, all the machines were assembled on imported components (processors, video cards), but domestic engineering solutions were used. And after 2014, we see a loss of interest, the degradation of the entire system began,” says Sergei Abramov.
Recall that in 2014, the Russian Federation began the first phase of its invasion of Ukraine by annexing Crimea. This was the first blow that Russia dealt to its relations with the rest of the world, and although cooperation continued until the start of Russia’s full-scale invasion of Ukraine on February 24, 2022, Crimea became the point of no return.
Between “Lomonosov” for Moscow State University and “Baikal” for the Ministry of Internal Affairs. The Story of Vsevolod Opanasenko and T-Platforms
The founder of the then leading domestic supercomputer company in Russia, T-Platforms, Vsevolod Opanasenko was arrested on March 26, 2019. During this time, the entire group of companies belonging to Opanasenko has been completely destroyed. Although he is no longer in jail or under house arrest, he is still involved in that criminal case. Despite the fact that we are talking about events that happened five years ago, at least a dozen of T-invariant’s interlocutors refused to discuss his case even anonymously. Including those who know the entrepreneur well and have been communicating with him personally all these years.
Vsevolod Opanasenko in court. Photo: https://www.kommersant.ru
Opanasenko is currently not prohibited from communicating with the press, but he himself has completely stopped doing so a long time ago. Although previously he was always a welcome guest of all leading publications (the best way to get acquainted is a detailed interview with Vedomosti in 2015, given on the wave of success from Julich; Vedomosti in 2015, given on the wave of success from Julich; problemswill only begin next year).
Reference
Vsevolod Opanasenko. Born April 12, 1972 in Moscow.
Education: 1995 — MATI — Tsiolkovsky Russian State Technical University, majoring in “engineer-technologist”; 2000 — Financial Academy under the Government of the Russian Federation, majoring in “securities market”.
Founded the company “T-Platforms” in 2002. The company gained international fame due to the development of the supercomputer “Lomonosov” in 2009. Its capacity was 1.7 petaflops (the number of floating-point operations per second). Lomonosov took 12th place in the world ranking. Over the entire period, six of the company’s solutions were included in the TOP-500 ranking of the most powerful computing complexes in the world (the undisputed leader among Russian companies). Since 2012, VEB has owned a blocking stake in T-Platforms OJSC. Since 2019, the company has not actually functioned, the bankruptcy process has begun, in 2021 the procedure was completed, but this did not prevent the United States from including it in its sanctions lists on March 31, 2022.
However, the first inclusion of the “T-Platform” in the US SDN lists occurred back in 2013. Then, through the efforts of diplomats and Russian officials, this problem was able to be solved. Interestingly, the reason for those sanctions was a successful contract to supply a supercomputer to the State University of New York in Stony Brook, where the initiator of T-Platform’s involvement was the famous chemist and crystallographer Artem Oganov, who now works in Russia and is known for his support of the invasion of Ukraine. The company was suspected of supplying computing systems to third world countries under sanctions and of collaborating with nuclear weapons manufacturers. “Within a year, the turnover on new deals dropped to zero, suppliers did not communicate with us and did not sell anything <…> We worked for lawyers throughout 2013,” said in an interview with Opanasenko.
In 2007, together with Sergei Abramov, Opanasenko received the Russian Federation Government Prize in Science and Technology for developing “design and software documentation, preparing industrial production and producing samples of high-performance computing systems (supercomputers) of the SKIF Series I and Series II family” (this was a long-term major project of the union state of Russia and Belarus). In 2015, together with Sergey Abramov and Alexander Moskovsky (RSK company, another market leader), he received the RAS Prize for a series of scientific papers on a single topic: “Development and implementation of a series of Russian supercomputers with cluster architecture.”
The lawyers from DINS Legal Group are still silent about the details of his criminal case (they did not respond to T-invariant’s request). At the beginning of 2019, this event stirred up the entire Russian business sector, which had not yet recovered from the «Baring Vostok case». Already in March, the Investigative Committee of the Russian Federation opened a criminal case against Opanasenko. He was accused of allegedly convincing the head of the Department of Information Technologies of the Ministry of Internal Affairs, Alexander Alexandrov, to help T-Platforms win a tender for the supply of computers to the Ministry of Internal Affairs departments. The policeman himself was arrested. Then, the famous lawyer Anatoly Kucherena, the secretary of the Public Chamber (at that time) Valery Fadeev, and the business ombudsman (2012-2022) Boris Titov spoke out in support of the “Russian Jobs” sent a letter to President Vladimir Putin, in which he asked to check how justified the criminal prosecution of the businessman was. Titov attached 78 sheets of documents to the letter, including the results of independent examinations (see the post) on Boris Titov’s Facebook account about this).
Current videos about science during the war, interviews, podcasts and streams with famous scientists – on the T-invariant YouTube channel. Become our subscriber!
Although the “Russian Jobs” became famous for supercomputers, he also earned money from quite trivial government orders. The background story goes back to the mid-2010s. In 2016, the Ministry of Internal Affairs announced that it would administer theoretical driving license exams on domestic computers running on Baikal-T1 processors. To do this, the agency decided to purchase 9,348 Tavolga Terminal computers for 357 million rubles (according to the documentation of the electronic auction announced by the Ministry of Internal Affairs). According to the government procurement website, only T-Platforms participated in it, and their subsidiary Baikal Electronics developed the processor of the same name. There was no competition: government procurement can be carried out from the sole supplierif his application was the only one and met the customer’s requirements. In October 2017, the company filed a lawsuit againstmvd MVD. The bill is almost 300 million rubles. Under such circumstances, the “largest government purchase of PCs on Russian processors” took place, which was then actively reported in the media “T-platforms”. The price of one “automated workstation” was 38,200 rubles, the company then emphasized that “hundreds of thousands” of Russians would be able to take the driving test on computers every year and that this was a good opportunity to advertise Russian computers to domestic users. All technical specifications of the Tavolga Terminal were provided by the CNEWS website.
How did they get around Putin’s amendments?
According to investigators, the Tavolga Terminal did not meet technical requirements and had “irreparable defects.” The Investigative Committee of Russia believes that Opanasenko knew about this, but persuaded Aleksandrov to provide T-Platforms with an advantage in the tender. The damage was assessed simply: exactly the amount of the contract, 357 million rubles. DINS Legal lawyers argued that Opanasenko and his company did not need any preferences by 2019, and, in addition, there were simply no other “computers with a domestic processor.” Titov wrote the same thing in a letter to Putin.
Whether the computers met the technical specifications or not is a subject for trial in a civil or arbitration court, but not a reason for a criminal case, Opanasenko’s lawyer Dmitry Lyashkov said at the time. The defense emphasized: Tavolga Terminal is used for traffic police exams, and the inspectors working with them are happy. Secret of the Company publication referringattention: the case was opened not under the article “fraud”, but under article No. 285 (“incitement to abuse of office”).
The article is incorrectly charged to Vsevolod Opanasenko, the head of the Business Protection Association, Alexander, claimed at the time. Khurudzhi.
It allows only public sector officials to be taken into custody, and Opanasenko is a businessman. According to the defense attorney, this helped the initiators of the criminal case to circumvent the “Putin amendments” to the Criminal Procedure Code, according to which suspected businessmen should be sent not to a pretrial detention center, but under house arrest. It is more difficult for the business ombudsman and other defenders of entrepreneurs’ rights to support those who are thus being tried under a non-economic article.
Between a pretrial detention center and Rostec’s “sharashka”
Opanasenko is not only the founder of a group of companies, the creator of top supercomputers, a man who was able to bring his business to the international market (in such a complex industry tied to world politics, the interests of the military and special services). According to most of T-invariant’s interlocutors, he is also a talented chief architect of all the company’s engineering projects.
In 2019, people who observed him in the pretrial detention center claimed: “He’s holding up well, he doesn’t complain about the regime.” And in 2024, all market participants say that the criminal case, which has been going on for five years, has not broken the businessman. “He’s back in the game, he has new projects, he’s back doing what he does best: assembling effective engineering teams, not forgetting about the result, product and profit,” one of the top managers in the industry, who has been closely acquainted with Opanasenko for a long time, told T-invariant.
JURECA supercomputer from T-Platform. Photo: https://sdelanounas.ru
“Seva is very resilient, he didn’t break down. He is a professional entrepreneur, and this is like a professional athlete. He is back in the game. Opanasenko has a reputation as a person who knows how to gather and motivate the best engineers in one place, and everyone is fighting for this now, including the guys on Frunzenskaya Embankment and on Usacheva. Components can always be bought, but not everyone can make a car,” another market participant tells T-invariant (these are the addresses of the Russian Ministry of Defense and the Rostec State Defense Corporation, respectively).
According to one version, it was the Rostec State Corporation, as well as the government officials and security officials who supervise it, that decided to “harass” Vsevolod Opanasenko, and the criminal case on the supply of computers to the Ministry of Internal Affairs became just a convenient pretext. T-invariant’s interlocutors talk about this, and they wrote about this in the comments under Boris Titov’s posts in defense of the businessman. After this question, former top managers of the subsidiaries of the state corporation Rusnano (which had a 25% stake in the company Baikal-Electronics, which was also associated with Opanasenko) hung up in conversations with a T-invariant correspondent.
Only the reasons why the entrepreneur allegedly became the target of persecution differ. The former top manager of one of the state corporations mentioned at the beginning of the text believes that Opanasenko “did not sell his business to the right state people associated with Rostec.” “Maybe he just dug in his heels, and he is a stubborn guy, or maybe they did not agree on the price,” he suggests.
“He was punished based on the sum of factors. Some big bosses promised the most important boss domestic computers, domestic processors, but he did not like the result. Opanasenko became the scapegoat. His problems began to accumulate long before the arrest, and competitors could have taken part in the case,” one of the market participants tells T-invariant.
Not a “sharashka”, but “on the hook”
And although Vsevolod Opanasenko formally returned to work a long time ago, his case is not closed yet. Now he is “on the hook”. There are several defendants in his case, one of them filed an appeal, which has not even been considered yet. Therefore, the courtthe debt decision has not yet entered into force.
“This is certainly not a sharashka of the 30s-40s, but rather he was simply “suspended”. It is unknown how long the case will take to complete. Whether they will come up with some more economic crimes is a big question. You could say that now he is “working off”. Both Opanasenko himself and many cool engineers and programmers are now working behind the fence,” one of the businessman’s colleagues explains vaguely. According to T-invariant’s interlocutors, this is about the former Solnechnogorsk Instrument Plant (SPZ), which is now called the Center for Supercomputer Modeling (CSM). The publication’s sources do not say how many organizations are part of the Center, what machines work there, and recommend not to trust it fragmentary public data which are periodically voiced by officials and employees of the state corporation. Unlike civilian computers, supercomputers behind the fence of Rostec and Rosatom do not appear in the world TOP-500 or the Russian TOP-50, so there is no independent and objective data about them. The center’s website does not seek to provide details about the company and looks has not been updated for a long time.
However, it was with the contract with Rostec that T-Platforms began having problems in 2016, which eventually led to the collapse of the company after a criminal case appeared. In 2016, SPZ filed a case in the Arbitration Court against T-Platforms, demanding a penalty of 281.5 million rubles “for equipment that was not delivered and installed on time.” The case concerned a contract between the company and SPZ worth 1.24 billion rubles concluded in December 2014 — specifically for the creation and launch of a powerful supercomputer. In July 2017, a settlement agreement was reached, and the CSM website states that the center began operations in 2018. The center’s first client was the Federal State Unitary Enterprise State Research Institute of Aviation Systems. In addition to high-performance computing and modeling, which was initially the focus of the center’s creation, the center’s description now mentions data analysis, processing and storage, cloud computing, and neural network training among its main activities.
Rostec periodically publishes press releases about the work of its other supercomputers. Here, for example, is a description of a machine called “Center”: “The Central Research Institute of Precision Engineering (TsNIITochMash) of the Rostec State Corporation has put into operation the high-performance software and technology complex “Center”. The complex, developed by order of the Ministry of Industry and Trade of the Russian Federation, is intended for physical and mathematical modeling of full-scale tests of small arms and ammunition. Its capacity is more than 50 trillion operations per second, which places it in the category of supercomputers.”
“In 2024, it became clear that three of the most powerful machines would be working in Russia, there is simply no need for more. One will work on education and science tasks – this is the new supercomputer, which is now at Moscow State University. The second – within the perimeter of Rosatom, in Sarov. The third – at Rostec. As far as I know, it is within the perimeter of Rostec that Seva now works. He simply has no choice. This is a typical job “behind the fence”, and, as far as I know, many cool guys are now “employed” there. No one will tell you what they do there. You shouldn’t even look in that direction,” says one of T-invariant’s interlocutors.
“They explained to him: you screwed up, now work it out,” one of the businessmen, who also works in the Russian supercomputer industry, describes the nature of Vsevolod Opanasenko’s relationship with Rostec even more specifically.
T-invariant tried to contact Opanasenko and passed on requests for an interview through his friends, and also sent requests to his lawyers, PR specialists and other colleagues, but received no response.
In the following parts
Vsevolod Opanasenko and the company “T-Platforms” installed supercomputers “Lomonosov-1” and “Lomonosov-2” at Moscow State University. The first machine has long been dismantled, the second, according to sources interviewed by T-invariant at the university, works with varying success, often breaking down. However, a new supercomputer has recently appeared on Vorobyovy Gory.
T-invariant will tell how another supercomputer was secretly built at Moscow State University in an atmosphere of complete secrecy, who works on it (taking into account the total economic and technological sanctions) and why access to it is difficult for scientists and employees of other universities and research institutes. We will also answer other questions that specialists have about the MSU-270 supercomputer.
MSU-270 supercomputer. Photo: https://msu.ru
We will dedicate a separate article in the series to how Russia’s full-scale invasion of Ukraine has affected the industry and will tell you how trade wars and tense relations in the US-China-Taiwan triangle are affecting it.
This series of materials will also discuss who the Govorun supercomputer in Dubna talks to and what it talks about (it was assembled by another market leader, the RSC company), who Nikolai Kristofari was and why two supercomputers were named in his honor, whether Yandex was able to establish a technological chain between Taiwan and Sasovo. Separately, we will find out who Nebius is and how he came into being after the defeat of Russia’s leading technology company, as a result of which the country lost computing power at the level of the world’s TOP-20. And we will continue to follow the fate of Sergei Abramov, in whose defense American physicists spoke out (you can read all the materials about Abramov here).
You can support the work of T-invariant by subscribing to our Patreon and choosing a convenient donation size.
Text: Editorial Board of T-invariant
Editors 27.08.2024