Jekyll

We have identified an issue with the way lancache handles the CACHE_MEM_SIZE/CACHE_INDEX_SIZE environment variables. If you are running a cache with more than 2tb of CACHE_DISK_SIZE we suggest you immediately do the following

• Check the value of CACHE_MEM_SIZE/CACHE_INDEX_SIZE in your .env. We now recommend this should be ~250m per 1TB of cache space
• Update Lan Cache (./update-containers.sh if you are using the compose)

​

More Detail:

Over the last 24 hours we have been diagnosing an issue where some caches were not filling their allocated CACHE_DISK_SIZE and started purging data out after ~2.5TB. This led to the discovery of an issues within our documentation and regression handling.

Issue 1: CACHE_MEM_SIZE vs CACHE_INDEX_SIZE

In October we updated the slightly confusing environment variable CACHE_MEM_SIZE to the more accurate CACHE_INDEX_SIZE. In order to prevent this breaking auto update scripts we have a deprecation check which is supposed to copy the value of CACHE_MEM_SIZE into CACHE_INDEX_SIZE on boot. Unfortunately in one of the two places this check is performed the script was erroring so all caches using the legacy CACHE_MEM_SIZE were being overwritten with the default CACHE_INDEX_SIZE of 500m.

This issue has now been resolved and we are looking at how to improve the deprecation handling and avoid these duplication errors in the future

​

Issue 2: How much CACHE_INDEX_SIZE do you actually need

CACHE_INDEX_SIZE is the amount of memory allocated to nginx to store the index of cache data. Our original recommendation in the documentation allowed 1000mb which is enough to store 8 million files (or 8TB of cache data). At some point we reduced this recommendation to 500mb which we felt should still have been sufficient to handle an average 4tb cache.

This recommendation is based on the assumption that the cache is full of 1mb slices. What we realised as part of our investigations into the cache not filling sufficiently is that the distribution of file sizes within the cache is actually a reverse bell curve, with much larger than expected numbers of ~<1kb files in addition to the expected large numbers of 1mb slices.

Based on analysis of real world usage across all CDN’s, we have now adjusted the recommendation to 250m per 1TB of cache data which allows 2 million files (at an average file size of 500kb). This should cover the variance with some headroom to spare.

Hi everyone, We have some great news that some of you may have already seen.

As of today’s Steam Client update, there is now full support in Steam for a lancache. The way it works is by setting a DNS override for lancache.steamcontent.com pointing to the IP address of your cache server. The cache domains list at uklans/cache-domains has already been updated to include this and if you redeploy lancachenet/monolithic and lancachenet/lancache-dns containers, it will pick up the new hostname and work.

The Steam client will then use that cache server for all downloads. It will still make requests to the cache server using the hostname of the Valve content server to download from, but it will connect to your cache.

In addition, it will increase the maximum number of connections per IP from 4 to 32. This should solve the issue people have been seeing with slow initial downloads through lancache. It will also prevent the Steam Client from using Open Caching, which caused some downloads to not be cached correctly.

An example of it running from my local PC’s Steam client (on the beta before Christmas) - the content.log file contains:

[2019-12-21 17:15:14] Enabling local content cache at '::ffff:c0a8:1e80' from lookup of lancache.steamcontent.com.
[2019-12-21 17:15:14] Adding cache type 'LANCache' on host '::ffff:c0a8:1e80'
[2019-12-21 17:15:14] Got 30 download sources and 0 caching proxies via ContentServerDirectoryService::BYieldingGetServersForSteamPipe (4/0)
[2019-12-21 17:15:14] Created download interface of type 'SteamCache' (7) to host cache12-lhr1.steamcontent.com (cache12-lhr1.steamcontent.com)
[2019-12-21 17:15:14] Created download interface of type 'SteamCache' (7) to host cache34-lhr1.steamcontent.com (cache34-lhr1.steamcontent.com)
[2019-12-21 17:15:14] Created download interface of type 'SteamCache' (7) to host cache26-lhr1.steamcontent.com (cache26-lhr1.steamcontent.com)
[2019-12-21 17:15:14] Created ipv4-only http client
[2019-12-21 17:15:14] HTTP (SteamCache,238) - cache26-lhr1.steamcontent.com (0.0.0.0:0 / 0.0.0.0:0, host: cache26-lhr1.steamcontent.com): AuthenticateDepotID (1521) - Success!
[2019-12-21 17:15:14] HTTP (SteamCache,238) - cache26-lhr1.steamcontent.com (192.168.30.128:80 / 192.168.30.128:80, host: cache26-lhr1.steamcontent.com): ::ffff:c0a8:1e80/depot/1521/manifest/6898858218219951821/5 - received 200 (OK) HTTP response
[2019-12-21 17:15:14] HTTP (SteamCache,43) - cache34-lhr1.steamcontent.com (192.168.30.128:80 / 192.168.30.128:80, host: cache34-lhr1.steamcontent.com): CDepotHTTPDownloadInterface::BYldTestConnection - Server response 'OK' (200)

I can’t thank Valve enough for all their help with this, they’ve been amazingly supportive of getting these changes in to Steam. They’re really interested in all your feedback about it, so if you have any feedback or comments on how well it’s working for you - stories, case studies, problems, statistics, etc. then please post them here or in our issue on Github

Please also spread the word about this to anyone you think might be interested in this.

As many of you know I’m one of the small team of maintainers on the steamcache/* docker images (along with @mintopia @GotenXiao @Lepidopterist @JasonRivers and @Astrolox). For some time now we have felt that we have long outgrown our steamcache roots and perhaps it was time to rebrand to the generic gaming cache which we now provide. After much consideration and some discussions with Multiplay and ChurchNerd we are proud to rebrand with immediate effect to LanCache.Net

All docker images have been updated and are available at https://cloud.docker.com/lancachenet/ we will continue to keep the steamcache images updated in line with these for the moment to ease migration but it’s well worth switching your images at the next opportunity.

Now, onto the REALLY big news!

The LanCache.Net and UKLans.net teams have been in discussions with the store team at Epic Games for the last few months. We are incredibly pleased to announce that with immediate effect Epic Games have migrated their CDNs and client to download over HTTP instead of HTTPS. This means those using our lancache-dns and monolithic containers will immediately be able to cache all games from the Epic Store!

This is still in beta testing right now (Pass -E DISABLE_EPIC_GAMES to the dns container if you wish to disable the EPIC store caching for the time being) but initial testing results are positive so we wanted to share this with you all as soon as possible. If you encounted any issues with epic then do raise an issue over at https://github.com/lancachenet/monolithic/issues/new

Huge thanks everyone over at Epic Games for working with us to a solution on this, the rest of the LanCache.Net team and of course everyone who has contributed issues, feedback, issues and pull requests!

Happy Gaming

@VibroAxe

Note: Most containers retain their original names so steamcache/monolithic becomes lancachenet/monolithic. The only exception to this is steamcache/steamcache-dns which has become lancachenet/lancache-dns