5

When updating a game on Steam, there's 2 metrics per game that the client is keeping track of: how much was downloaded and how much was written to disk. I know compression exists and there are some edges cases like <4KB files taking up 4KB on disk (or whatever your sector size is) so it's rare that these 2 numbers are exactly the same. But sometimes these numbers are EXTREMELY different. I'm currently updating Halo Infinite and this is what I see in the client:

Halo Infinite download progress. 624.9MB will be downloaded. 35.3GB will be written to disk.

Unless the game contains GIGS of redundant data that compresses incredibly well, I'm not sure how a ~600MB download could result in ~35GB of modifications to disk. I don't think Steam is doing anything malicious, but I'm very curious about what's happening here. I see this as a learning opportunity. My wildest guess is that maybe a small utility is downloaded that when run will modify large pieces of data like modifying 3D model geometry for whatever reason. It's a complete shot in the dark that's probably wrong.

I'm familiar with a lot of the optimizations that other software distribution platforms use. There may be multiple features working together to bring the download size as low as possible. Compression, only download what changes, only download patches for large files, but in my mind these don't add up to the 70 fold increase between what's downloaded and what's modified. I've never published a game on Steam before so I don't know what options developers have available to them. Is there any deployment documentation that could explain this?

Improve this question
1
  • 3
    At a guess, Steam is counting the entire filesize of the files it's patching. Unreal Engine games in particular tend to consist of a small number of very large files.
    – Powerlord
    Mar 28 at 23:08

1 Answer 1

Reset to default
4

The following is based on a combination of a SteamDB Blog entry partly about Steam's download system and the beginning of the official Steamworks documentation of SteamPipe (Steam's content delivery system) as suggested at the end of that blog post.

The content of a game is split into roughly one megabyte chunks, and the chunks are then compressed with LZMA and encrypted using AES with a 256-bit key. A list of info for each file - encrypted name, size, a hash, and flags (e.g. file or directory, executable), and chunk metadata - is stored in a depot manifest, which identifies a single version of some downloadable content.

Chunk metadata is a chunk id, adler32 checksum, offset in the file, and compressed and original sizes. When creating the update that users will download, SteamPipe searches to find unchanged chunks that can be directly copied, and so only requires users to download the new or changed chunks to update their locally-installed games. A delta update, e.g. only the changes.

Lots of talk about pack files in the SteamPipe documentation. Technical implementation details that are largely per-game. Some potential issues mentioned are unnecessary pack file compression without following SteamPipe guidelines, pre-SteamPipe encryption, and how some versions of Unreal Engine use asset "padding alignment" in pack files which can unnecessarily bloat update sizes. Extremely shortened: if pack files are not well managed it can greatly increase download sizes, and cause longer install times if a user has slow storage.

Not part of the explanation of behind the scenes, but a fun addition because of it's relevance all around.

Looking at Halo: Infinite's update history on SteamDB it looks like they very conveniently heavily updated some pack files around this time. Specifically, they seem to have split them into smaller chunks - note how there were originally only 8 for ...mpspartan... (red, deleted) but that increased to 40 files (green, modified/added). This is specifically mentioned in the SteamPipe docs as something to avoid unless it will help in the long run by decreasing users load times and/or update sizes.

Old files: 755.18 + 763.49 + 733.44 + 778.68 + 873.09 + 804.82 + 838.43 + 932.48 = 6479.610000000001 KiB, or about 6.17 MiB.

New files: 139.59 + 44.83 + 48.05 + 125.81 + 95.49 + 113.48 + 43.22 + 44.10 + 119.11 + 95.84 + 117.32 + 41.11 + 41.64 + 132.59 + 95.18 + 133.18 + 43.98 + 45.35 + 133.77 + 102.51 + 108.78 + 41.05 + 44.97 + 119.25 + 86.03 + 110.88 + 38.14 + 40.43 + 124.33 + 81.88 + 117.59 + 39.45 + 40.82 + 121.26 + 85.08 + 104.35 + 39.17 + 43.14 + 122.72 + 82.74 = 3348.2099999999996 KiB, or about 3.19 MiB.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .