Hailey on Nostr: I remain unconvinced that docker layers are a good abstraction. Building an image ...
I remain unconvinced that docker layers are a good abstraction.
Building an image that uses another image as a base? Sure, that makes sense, but keeping all those layers around and exposing them to the user as a domain concept does not.
There's an ongoing runtime perf cost to supporting them, and they're just not that effective when it comes to deduplicating image contents. Consider your `bundle install` layer, yeah you can reuse it between app versions that don't bump any gems, but the moment you bump even one gem, you're paying hundreds of MB if not close to a GB for that bump.
I keep thinking about this paper (
https://arxiv.org/pdf/2305.13162.pdf) out of the AWS Lambda team where they mention flattening layers into a single ext4 with a deterministic extent layout. Deduplicating 512 KiB chunks of this image turns out to be a lot more effective for them than layer-based deduplication ever was, plus it enables image lazy loading in a way that layers simply can't achieve.
Published at
2023-07-18 03:39:06Event JSON
{
"id": "f91d41cafe5c9f6db911f27f3600e94458707f4b42ba7450f445201c8ef9cf6f",
"pubkey": "ba435c386a4b42d56d225e3662979d7904c2cd907150f293d0cd8a6019121f80",
"created_at": 1689651546,
"kind": 1,
"tags": [
[
"mostr",
"https://hails.org/users/hailey/statuses/110733003724080733"
]
],
"content": "I remain unconvinced that docker layers are a good abstraction.\n\nBuilding an image that uses another image as a base? Sure, that makes sense, but keeping all those layers around and exposing them to the user as a domain concept does not.\n\nThere's an ongoing runtime perf cost to supporting them, and they're just not that effective when it comes to deduplicating image contents. Consider your `bundle install` layer, yeah you can reuse it between app versions that don't bump any gems, but the moment you bump even one gem, you're paying hundreds of MB if not close to a GB for that bump.\n\nI keep thinking about this paper (https://arxiv.org/pdf/2305.13162.pdf) out of the AWS Lambda team where they mention flattening layers into a single ext4 with a deterministic extent layout. Deduplicating 512 KiB chunks of this image turns out to be a lot more effective for them than layer-based deduplication ever was, plus it enables image lazy loading in a way that layers simply can't achieve.",
"sig": "d90708a1b0ac1ec2d1a6ffae5a60ab733ea4c57fc19a9cc6d7245edde27703c208823670974354bbcd67a3c76d18ab02cf69ff403ea61eb0ff204569f42c8bce"
}