Best practices for setting up Git LFS

[redkite]

Godot Version

4.x

Question

What are the best practices for setting up Git LFS for your Godot project’s repository? More specifically, what exactly should the LFS be tracking?

TL;DR below, but would appreciate reading the whole post to understand my thought process. :>

So far I have seen two ways of tracking your binary files.

A: Creating an assets folder which contains all your binaries and then tracking this directory as a whole.

B: Adding each and every binary file extention to the .gitattributes file from .png, .mp3, .fbx all the way to Godot’s own binaries like .scn and .res similar to this post.

Both options have their own advantages and disadvantages.

If you have a well organized project structure, then option A is a simple way of tracking all large files by just adding one directory to the .gitattributes file. However, files like .scn and .res are not going to be tracked if they are saved in a different folder just for resources.

Option B on the other hand allows a more precice tracking regardless of where files are saved within the project structure. You just have to remember to add all the necessary file extentions to your .gitattribute file and not forget in case you some day, for some reason, do end up having to make an obscure file like .multimesh. XD

Also what about some text-based assets?

Like the .import files that Godot generates with every imported asset. Using option A means that those will be tracked too. They are text-based so technically they don’t need to be tracked. But on the other hand, they are closely linked to the assets they belong to. And it’s not like you edit those a whole bunch either. So should they be tracked as well?

And what about .svg files. While being images, they are text-based as well. So same could be said about them too: They aren’t edited much; might as well track them too, right?

Thank you for reading!

Maybe I’m being a little too overcautious with this, but I just wanna set up my LFS the right way in order to avoid problems with it later.

Let me know how you set up Git LFS for your larger projects. If you use Git at all that is, considering it’s not the greatest for game projects.

TL;DR

Track the entire assets folder or pick out file extentions individually?
Also, .import files: Track them or not?

[jtbourke]

When I first started using git and Godot, which wasn’t that long ago, I was misled by some well intentioned forum posts, and I asked similar questions. What I found is that LFS adds overhead to each tracked file and adds some complexities.

What I recommend you do is ignore LFS until you have a reason for it.

I do not think it will pay off to structure your project around LFS, unless you have a lot of large files that are being revised a lot.

What are the best practices for setting up Git LFS for your Godot project’s repository? More specifically, what exactly should the LFS be tracking?

Use it to track individual files that are both 1) large, and 2) changing often.

A: Creating an assets folder which contains all your binaries and then tracking this directory as a whole.
B: Adding each and every binary file extention to the .gitattributes file from .png, .mp3, .fbx all the way to Godot’s own binaries like .scn and .res similar to this post.

Option A will be simple from a git perspective, but will require a lot of discipline in how you add files. Option B means that a lot of files will be tracked that don’t need to be tracked. Take Option C which is to track only the individual files that justify the overhead and complexity.

Also what about some text-based assets?

Don’t LFS track these at all. Don’t LFS track text files at all, don’t track files just because they are binary, and don’t worry about large files unless they are changing a lot.

You can always add LFS tracking to deserving files later, so you do not need to consider this at all until you notice performance issues. Worst case if you have a large binary file that is getting a changed a lot, the initial clone will take a long time. That isn’t the end of the world. Don’t optimize early here.

Of course, if you are using Github or pretty much any git provider, they will have a limit on how large individual files can be. This is probably the number 1 actual reason people use LFS! So you will possibly be forced to LFS track some files. If that’s the case, just let the provider reject your push and it can tell you which files need to be tracked.

Hope that helps.

Jim

[redkite]

Thank you for your reply! Sorry I didn’t respond until now. ^^’

@jtbourke

Use it to track individual files that are both 1) large, and 2) changing often.

I suppose that makes sense. Don’t make it more complicated than it has to be. But in early stages of development, how do I even know if a file will be changed a lot? Like yeah, there will be early placeholder files like 3D models for characters or maps where I know they will get revised a lot. But with other files like audio or maybe art, who knows how many times I will have to adjust them?

@jtbourke

Take Option C which is to track only the individual files that justify the overhead and complexity.

Would that not be a bit tedious to do? At least from a Git perspective, as you said earlier. Tracking individual files could make things hard to overview and just get plain messy as the project and repository complexity grows.

And what about if I want to move a file within the project folder from one directory to another? Or what if I want to rename it to a different name later? The path would change so would I need to adjust the tracked path as well? Or is Git smart enough to change this on its own?

@jtbourke

You can always add LFS tracking to deserving files later, so you do not need to consider this at all until you notice performance issues.

Is that true? I was under the impression that once you commit a file to the repository, you cannot go back and LFS track it retrospectively. Once it’s out it’s out like this forever; or at least hard to change that. Maybe I am wrong here though. Would I just need to git lfs track [path] it?

@jtbourke

Worst case if you have a large binary file that is getting a changed a lot, the initial clone will take a long time.

And also, as far as I understand, tracked files also reduce the size of your clones as the cloning process will not have to grab every binary file for every commit and branch there is; instead just grabbing the files once and then adding the hook. Is that not a point for option A, preventing your local clone from getting really huge in size?

---

Either way, thank you for your input! I appreciate it. :>
I would love to hear more takes on this from other people as well. So far I have had every person tell me something different on how to proceed so I would like to hear more.

[gertkeno]

@redkite

Is that true? I was under the impression that once you commit a file to the repository, you cannot go back and LFS track it retrospectively. Once it’s out it’s out like this forever; or at least hard to change that. Maybe I am wrong here though. Would I just need to git lfs track [path] it?

The previous blob versions will persist inside the repo, but LFS versions thereafter can replace it. You can also remove from LFS at any time.

For reference LFS replaces the blob file with a short version/id string. When pushing a change it only pushes the version/id to git and it uploads the blob the a separate LFS server. When pulling it looks at the version/id string in git and downloads the blob from the LFS server.

This is done transparently through git hooks, so you see the real files and git only sees the version/ids.

[jtbourke]

@redkite

But in early stages of development, how do I even know if a file will be changed a lot? Like yeah, there will be early placeholder files like 3D models for characters or maps where I know they will get revised a lot. But with other files like audio or maybe art, who knows how many times I will have to adjust them?

You can’t know that, which is why you shouldn’t try to.

Would that not be a bit tedious to do?

Yes, it can be tedious. LFS is a work-around to a shortcoming of git. That’s why it has these annoyances.

Looking back on this as a former (and I guess still current, but I’m working on that) heavy user of perforce, I feel like git’s problems with binary files were overblown by the marketing teams of competing paid products. If you just run git and ignore LFS it will probably be fine for you.

And what about if I want to move a file within the project folder from one directory to another? Or what if I want to rename it to a different name later? The path would change so would I need to adjust the tracked path as well? Or is Git smart enough to change this on its own?

Huh. I like that question. I’ve never tried it but I think I know what will happen. You should test this out and let us all know.

The files are LFS tracked with an entry in the .gitattributes file. You can use a full path name or just a file name when you identify files as LFS tracked in .gitattributes. My guess is that the .gitattributes reference will not be updated when you rename a file (or move it). So you would have the original entry still there which could still work depending on how you referenced the file. Or, if the file reference is no longer valid, you would have an orphaned reference in .gitattributes and the file would no longer be tracked. That’s my guess.

Is that true? I was under the impression that once you commit a file to the repository, you cannot go back and LFS track it retrospectively. Once it’s out it’s out like this forever; or at least hard to change that. Maybe I am wrong here though. Would I just need to git lfs track [path] it?

Look up “git lfs migrate”

And also, as far as I understand, tracked files also reduce the size of your clones as the cloning process will not have to grab every binary file for every commit and branch there is; instead just grabbing the files once and then adding the hook. Is that not a point for option A, preventing your local clone from getting really huge in size?

Yes, the whole point of LFS is to reduce the size of clones. If you don’t care about the time (or disk space) it takes to clone then there isn’t much need for LFS.

Is that not a point for option A, preventing your local clone from getting really huge in size?

Yes, that’s the reason to use it.

Jim