Need To Move 1.2 Exabytes Across the World Every Day? Just Effingo
An anonymous reader shares a report: Google has revealed technical details of its in-house data transfer tool, called Effingo, and bragged that it uses the project to move an average of 1.2 exabytes every day. As explained in a paper [PDF] and video to be presented on Thursday at the SIGCOMM 2024 conference in Sydney, bandwidth constraints and the stubbornly steady speed of light mean that not even Google is immune to the need to replicate data so it is located close to where it is processed or served.
Indeed, the paper describes managed data transfer as "an unsung hero of large-scale, globally-distributed systems" because it "reduces the network latency from across-globe hundreds to in-continent dozens of milliseconds." The paper also points out that data transfer tools are not hard to find, and asks why a management layer like Effingo is needed. The answer is that the tools Google could find either optimized for transfer time or handled point-to-point data streams -- and weren't up to the job of handling the 1.2 exabytes Effingo moves on an average day, at 14 terabytes per second. To shift all those bits, Effingo "balances infrastructure efficiency and users' needs" and recognizes that "some users and some transfers are more important than the others: eg, disaster recovery for a serving database, compared to migrating data from a cluster with maintenance scheduled a week from now."
<a href="http://twitter.com/home?status=Need+To+Move+1.2+Exabytes+Across+the+World+Every+Day%3F+Just+Effingo%3A+https%3A%2F%2Fhardware.slashdot.org%2Fstory%2F24%2F08%2F06%2F1650258%2F%3Futm_source%3Dtwitter%26utm_medium%3Dtwitter"; rel="nofollow"><img src="https://a.fsdn.com/sd/twitter_icon_large.png"></a>;
<a href="http://www.facebook.com/sharer.php?u=https%3A%2F%2Fhardware.slashdot.org%2Fstory%2F24%2F08%2F06%2F1650258%2Fneed-to-move-12-exabytes-across-the-world-every-day-just-effingo%3Futm_source%3Dslashdot%26utm_medium%3Dfacebook"; rel="nofollow"><img src="https://a.fsdn.com/sd/facebook_icon_large.png"></a>;
https://hardware.slashdot.org/story/24/08/06/1650258/need-to-move-12-exabytes-across-the-world-every-day-just-effingo?utm_source=rss1.0moreanon&utm_medium=feed
at Slashdot.
https://hardware.slashdot.org/story/24/08/06/1650258/need-to-move-12-exabytes-across-the-world-every-day-just-effingo?utm_source=rss1.0mainlinkanon&utm_medium=feed
Slashdot (RSS Feed) /
npub1rk…d8w8z
2024-08-06 16:50:00
![https://a.fsdn.com/sd/twitter_icon_large.png"></a>](https://a.fsdn.com/sd/twitter_icon_large.png"></a>){kind=link}
![https://a.fsdn.com/sd/facebook_icon_large.png"></a>](https://a.fsdn.com/sd/facebook_icon_large.png"></a>){kind=link}
Author Public Key
npub1rk3j5fc4ew5w9zd7kx5zfqt04tew0kan9swrr63ctn6k8wp8f65qwd8w8zPublished at
2024-08-06 16:50:00Event JSON
{
"id": "91304f0fd3cab6ce39874bb70dcd43491fb9e9a9dad78f944e3feeda7bc75741",
"pubkey": "1da32a2715cba8e289beb1a824816faaf2e7dbb32c1c31ea385cf563b8274ea8",
"created_at": 1722963000,
"kind": 1,
"tags": [
[
"t",
"storage"
],
[
"proxy",
"http://rss.slashdot.org/Slashdot/slashdot#https%3A%2F%2Fhardware.slashdot.org%2Fstory%2F24%2F08%2F06%2F1650258%2Fneed-to-move-12-exabytes-across-the-world-every-day-just-effingo%3Futm_source%3Drss1.0mainlinkanon%26utm_medium%3Dfeed",
"rss"
]
],
"content": "Need To Move 1.2 Exabytes Across the World Every Day? Just Effingo\n\nAn anonymous reader shares a report: Google has revealed technical details of its in-house data transfer tool, called Effingo, and bragged that it uses the project to move an average of 1.2 exabytes every day. As explained in a paper [PDF] and video to be presented on Thursday at the SIGCOMM 2024 conference in Sydney, bandwidth constraints and the stubbornly steady speed of light mean that not even Google is immune to the need to replicate data so it is located close to where it is processed or served. \n\nIndeed, the paper describes managed data transfer as \"an unsung hero of large-scale, globally-distributed systems\" because it \"reduces the network latency from across-globe hundreds to in-continent dozens of milliseconds.\" The paper also points out that data transfer tools are not hard to find, and asks why a management layer like Effingo is needed. The answer is that the tools Google could find either optimized for transfer time or handled point-to-point data streams -- and weren't up to the job of handling the 1.2 exabytes Effingo moves on an average day, at 14 terabytes per second. To shift all those bits, Effingo \"balances infrastructure efficiency and users' needs\" and recognizes that \"some users and some transfers are more important than the others: eg, disaster recovery for a serving database, compared to migrating data from a cluster with maintenance scheduled a week from now.\"\n\u003ca href=\"http://twitter.com/home?status=Need+To+Move+1.2+Exabytes+Across+the+World+Every+Day%3F+Just+Effingo%3A+https%3A%2F%2Fhardware.slashdot.org%2Fstory%2F24%2F08%2F06%2F1650258%2F%3Futm_source%3Dtwitter%26utm_medium%3Dtwitter\" rel=\"nofollow\"\u003e\u003cimg src=\"https://a.fsdn.com/sd/twitter_icon_large.png\"\u003e\u003c/a\u003e\n\u003ca href=\"http://www.facebook.com/sharer.php?u=https%3A%2F%2Fhardware.slashdot.org%2Fstory%2F24%2F08%2F06%2F1650258%2Fneed-to-move-12-exabytes-across-the-world-every-day-just-effingo%3Futm_source%3Dslashdot%26utm_medium%3Dfacebook\" rel=\"nofollow\"\u003e\u003cimg src=\"https://a.fsdn.com/sd/facebook_icon_large.png\"\u003e\u003c/a\u003e\n\n\n\nhttps://hardware.slashdot.org/story/24/08/06/1650258/need-to-move-12-exabytes-across-the-world-every-day-just-effingo?utm_source=rss1.0moreanon\u0026utm_medium=feed\n at Slashdot.\n\nhttps://hardware.slashdot.org/story/24/08/06/1650258/need-to-move-12-exabytes-across-the-world-every-day-just-effingo?utm_source=rss1.0mainlinkanon\u0026utm_medium=feed",
"sig": "29f10d1a25879d89ca3775590e194127502467fb73ee504d765ccb5db96f991bb27945c1cb8116f216c53d56e816febf94b97bfe88100fbb824c44f97f9a1609"
}