Troubles with Saving and Loading Dictionary of Resources

Help
,
1

Hello, I am using Godot 4.4.1.stable and (attempting to be) using FileAccess as my means of saving and loading data. I do not want to use JSON.

In my SAVEDATA class (which extends Resource), I have an @export Dictionary of 3 elements, called Data. These elements are themselves another class (which also extends Resource) PerLevelSaveData. PerLevelSaveData contains 4 @export variables:

  • ScoreNormal (‘normal’ game mode hiscore; int)
  • TimeAttackBest (‘time attack’ game mode best time; float)
  • ScoreHard (‘hard’ game mode hiscore; int)
  • GemstonesCollected (one-time collectibles, all named “Gemstone” with a numerical index as a suffix, added to the dictionary with its collection state (true/false). This is done a maximum of 255 times; Dictionary (string : bool))

I initially attempted to use ResourceSaver.save() and ResourceLoader.load() to save and load SAVEDATA, creating an object that contains a dictionary of 3 instances of PerLevelSaveData. This resulted in ResourceSaver.save() failing, returning error code 15.
Next, I tried using FileAccess to save and load. Saving Data (or another instance of it) yielded a save file appearing. However when attempting to load it, though it recognised it as a dictionary containing 3 elements it would fail to give me any elements of that dictionary in a way that was readable. All elements were EncodedObjectAsID. Attempting to cast them to PerLevelSaveData (the expected type), the element would return as null.

My save function is as follows:

func Save(path : String):
    var fa = FileAccess.open(path, FileAccess.WRITE)
    fa.big_endian = true
    fa.store_var(Data)
    fa.close()

My load function is as follows:

func Load(path : String):
    var fa = FileAccess.open(path, FileAccess.READ)
    fa.big_endian = true
    Data = fa.get_var()
    fa.close()

This code should directly save and load a dictionary of PerLevelSaveData resources. I am not doing anything out of the ordinary with either the ResourceSaver/ResourceLoader nor the FileAccess method.
Attempting to use instance_from_id(Data[k]) (where k is the level key) still yields the dictionary elements being EncodedObjectAsID.
Lastly, the save file doesn’t actually seem to be saving PerLevelSaveData, as the save file is much smaller than expected.

What exactly am I missing or doing wrong ? How do I properly save my resources such that they can be loaded again ?

Thank you for any help. I’m new to Godot and this is my first project with it after developing in Unity for more than 10 years. Looking forward to making more use of Godot’s tools.

2

You should always use ResourceSaver for resource serialization. Show the code that caused error 15.

3

Hi,
I’ll keep in mind that I should use ResourceSaver instead, so long as I can get a working solution. There’s not (supposed to be) much to it. The code for using ResourceSaver looked like this:

func Save(path : String):
    take_over_path(path)
    print("ResourceSaver ERR Response: " + str(ResourceSaver.save(self, path, ResourceSaver.FLAG_SAVE_BIG_ENDIAN))) #this executes ResourceSaver.save() and returns an error code. Currently returns 15.

and the code for using ResourceLoader looked like this:

func Load(path : String):
    var load = ResourceLoader.load(path) as SaveData
    Data = load.Data

If you need to know how Data is being initialised, it’s like this:

@export var Data : Dictionary = {
    1 : PerLevelSaveData.new(),
    2 : PerLevelSaveData.new(),
    3 : PerLevelSaveData.new()
}

PerLevelSaveData was described in my previous post.

4

Why are you calling take_over_path() and what’s the value of path?

5

I read that saving can fail if a file already exists and take_over_path() acts as a way to ensure it’s overwritten. Perhaps I was mistaken and it’s not necessary ?

The value of path is user://data.sav. This points to the user directory and should save the file as data.sav.

6

Resource saver and loader identify resource/file type by file extensions. You can’t just use an arbitrary extension. That’s why you get the error. res or tres should work. You can get the full list of extensions for a resource class by calling ResourceSaver.get_recognized_extensions()

1 Like
7

Had no idea ! I’ll remember that in the future. Updating the path to user://data.res, the ResourceSaver and ResourceLoader now both work.

While this is a solution, I don’t think it is necessarily the solution.

I’ve been shown I can get my FileAccess version working if I enable full objects when saving and loading the file. This worked, but now the file is very bloated. Almost as big as the ResourceSaver version. I’m wanting to condense it down.

How should I set up the save function or resources to condense it down ? Obviously this would require using things like store_buffer() with a byte array to replace the dictionary or bool array in PerLevelSaveData, but with PerLevelSaveData still being a resource, I’m not quite sure how else to save its data in as flexible of a manner (yet).

I’ll keep working on this. Once it’s complete, I’ll follow up with my solution. If I don’t finish it, I’ll mark your answer as the solution and just eat the extra bytes. Thanks so much for your help so far ! Hope to get a solution I’m more than happy with.

8

Why? What makes you think it’s “too big”?

9

It’s currently around 7kb for 3 levels worth of data, a little over 2kb per level. Obviously the dictionary/array of bools is the biggest source of bloat, so I can convert it to a PackedByteArray and shrink it down that way. I just don’t know if the overhead of saving it as a Resource is necessarily worth it. Having a workaround will let me assess such.
Additionally, I’ve read that saving full objects (whether through ResourceSaver or FileAccess) opens an attack vector for arbitrary code execution, which I’d like to avoid if possible. I think I’m wanting to find a solution that uses FileAccess but doesn’t save full objects. If that’s simply not possible, I’ll just make the PackedByteArray and call it there :face_savoring_food:

10

Resources are convenient and flexible to use during the development. You can always switch to some other form of serialization later.

If you’re really worried about kilobytes and “attack vectors”, then pack booleans into bitstreams and make a full custom binary serialization system, where you define the file format and control every byte via store_var() and get_var(). It’ll require some work and constant updates to the system though, whenever you add a new property to your file format. If that’s something you’re enjoying doing - go for it. I’d much rather just use the built in resources system that automatically handles all of the dirty serialization gruntwork.

11

Yes, I’m just not quite sure where to get started in a custom binary serialisation. Any pointers would be appreciated.
Again, I’ll post an update with my solution once it’s complete. Thank you so much for your help thus far !

12

Look at some common binary file formats that have open specifications, to gets some ideas on how things can be arranged. For example GIF, which is a rather simple format with more or less fixed structure, or something like PNG, a chunked format that’s more complex but also more flexible.

In the end, the format will be highly dependent on the type and structure of the data you need to serialize. There’s really no one-size-fits-all solution. Otherwise there wouldn’t be so many binary file formats in existence. That’s in fact one of the main advantages of binary serialization - you can tightly tailor it to your very specific needs.

1 Like
13

That’s not big unless your users are going to be running your game on computers from the 80s.

Yes it does - but only if you as the game developer allow it. There have been multiple discussions on this forum about that subject in the past year.

You’ll have to code that yourself, making a lookup dictionary in your code so that you can store multiple bools in a single int.

14

Hey all ! Thank you for your insight and feedback. I’ve gone ahead and come up with a solution I’m happy with, so I will outline the solution(s) I’ve received.

Method 1: ResourceSaver and ResourceLoader

I learned an important note about this method. You have to save it with a .res or .tres file extension or else the saving fails. Thank you to @normalized for this revelation. Additional extensions can be found via ResourceSaver.get_recognized_extensions().
Here’s the code:

SAVEDATA.gd

func Save(path : String): #path is "user://save.res"
	ResourceSaver.save(self, path)

func Load(path : String): #path is "user://save.res"
	var load = ResourceLoader.load(path)
	Data = load.Data

Method 2: FileAccess with Full Objects

I learned that in my initial method, I was missing a flag to serialise full objects. This works basically the same as ResourceSaver.save() and contains a lot of the same data. Some variables get serialised differently, so be careful when retrieving them. This method came to me courtesy of @aeonofdiscord@icosahedron.website on the Mastodon network.
Here’s the code:

SAVEDATA.gd

func Save(path : String): #path is "user://save.dat" (or any arbitrary filename+extension)
	var fa = FileAccess.open(path, FileAccess.WRITE)
	fa.store_var(Data, true) #enables serialisation of full objects
	fa.close()

func Load(path : String) #path is "user://save.data" (or any arbitrary filename+extension)
	var fa = FileAccess.open(path, FileAccess.READ)
	Data = fa.get_var()
	fa.close()

Method 3: Writing Your Own Basic Serialiser

This is the method I went with and what works for me. I like nice, compact files, maybe some padding. I am limited by not knowing everything Godot has to offer, so this solution is a little crass, but it works ! Some pointers from @normalized and @aeonofdiscord@icosahedron.website helped me get here.
Here’s the code:

SAVEDATA.gd

func Save(path : String): #path is "user://save.dat" (or any arbitrary filename+extension)
	var leveldata : Array[PackedByteArray] = []
	var keys = Data.keys()
	for key in keys:
		leveldata.append(Data[key].Serialize())
	var fa = FileAccess.open(path, FileAccess.WRITE)
	for i in range(0, leveldata.size()):
		fa.store_buffer(leveldata[i])
	fa.close()

func Load(path : String): #path is "user://save.dat" (or any arbitrary filename+extension)
	var keys = Data.keys
	var fa = FileAccess.open(path, FileAccess.READ)
	for key in keys:
	#I have 8 variables (ScoreNormal, TimeAttackbest, ScoreHard, the ints I'm storing bits into, and padding) all of which are 8 bytes long.
		(Data[key] as PerLevelSaveData).Deserialize(fa.get_buffer(8*8))
	fa.close()
	#I set the game's "active save file" here by setting it to SAVEDATA self

PerLevelSaveData.gd

Notes: GemstonesCollected is initialised to a Dictionary of bools with 256 entries, 0-255.

func Serialize() -> PackedByteArray:
	var ret : PackedByteArray
	ret.resize(8*8)
	var int0to63 : int = GemstonesCollected["Gemstone0"]
	var int64to127 : int = GemstonesCollected["Gemstone64"]
	var int128to191 : int = GemstonesCollected["Gemstone128"]
	var int192to255 : int = GemstonesCollected["Gemstone192"]
	for i in range(0, 256):
		if(i >= 192):
			int192to255 = int192to255 << 1
			int192to255 |= int(GemstonesCollected["Gemstone"+str(i)])
			continue
		if (i >= 128):
			int128to191 = int128to191 << 1
			int128to191 |= int(GemstonesCollected["Gemstone"+str(i)])
			continue
		if (i >= 64):
			int64to127 = int64to127 << 1
			int64to127 |= int(GemstonesCollected["Gemstone"+str(i)])
			continue
		if (i >= 0):
			int0to63 = int0to63 << 1
			int0to63 |= int(GemstonesCollected["Gemstone"+str(i)])
			continue
	ret.encode_u64(0*8, ScoreNormal)
	ret.encode_float(1*8, TimeAttackBest)
	ret.encode_u64(2*8, ScoreHard)
	ret.encode_u64(3*8, int0to63)
	ret.encode_u64(4*8, int64to127)
	ret.encode_u64(5*8, int128to191)
	ret.encode_u64(6*8, int192to255)
	ret.encode_u64(7*8, Padding)
	return ret

func Deserialize(buffer : PackedByteArray) -> PerLevelSaveData:
	ScoreNormal = buffer.decode_u64(0*8)
	TimeAttackBest = buffer.decode_float(1*8)
	ScoreHard = buffer.decode_u64(2*8)
	
	var int0to63 : int = buffer.decode_u64(3*8)
	var int64to127 : int = buffer.decode_u64(4*8)
	var int128to191 : int = buffer.decode_u64(5*8)
	var int192to255 : int = buffer.decode_u64(6*8)
	buffer.decode_u64(7*8) #standalone statement to move cursor to end of buffer
	for i in range(255, -1, -1):
		if(i >= 192):
			GemstonesCollected["Gemstone"+str(i)] = bool(int192to255 & 1)
			int192to255 = int192to255 >> 1
			continue
		if(i >= 128):
			GemstonesCollected["Gemstone"+str(i)] = bool(int128to191 & 1)
			int128to191 = int128to191 >> 1
			continue
		if(i >= 64):
			GemstonesCollected["Gemstone"+str(i)] = bool(int64to127 & 1)
			int64to127 = int64to127 >> 1
			continue
		if(i >= 0):
			GemstonesCollected["Gemstone"+str(i)] = bool(int0to63 & 1)
			int0to63 = int0to63 >> 1
			continue
	return self

Thanks again to everyone’s help, insight, and feedback. I learned a lot making this ! My save file is now a teeny tiny 192 bytes long, with padding. That’s 64 bytes per level ! Now that’s what I’m talking about. As always, if there’s ways to improve it, I’m open for suggestions.

Have a great one !

3 Likes
15

You can also use ResourceFormatSaver and ResourceFormatLoader to add your own file extension to ResourceSaver and ResourceLoader classes. This way you can use ResourceSaver.save() and ResourceLoader.load() to save or load your custom file to a resource.

ResourceFormatLoader

@tool
class_name MyResourceFormatLoader
extends ResourceFormatLoader


func _recognize(resource: Resource) -> bool:
	return resource is MyResource


func _get_recognized_extensions() -> PackedStringArray:
	return PackedStringArray(["custom_extension"])


func _load(path: String, original_path: String, use_sub_threads: bool, cache_mode: int) -> Variant:
	# Load the custom file using FileAccess and process the data
	# return MyResource

ResourceFormatSaver

@tool
class_name MyResourceFormatSaver
extends ResourceFormatSaver


func _recognize(resource: Resource) -> bool:
	return resource is MyResource


func _get_recognized_extensions(resource: Resource) -> PackedStringArray:
	return PackedStringArray(["custom_extension"])


func _save(resource: Resource, path: String, flags: int) -> Error:
	if resource is not MyResource:
		return Error.FAILED
	# Here you will save your resource object to your custom file
	# return Error.OK

Register And Deregister The Resource Formats

# You have to register and deregister the your custom resource format with the resource system. This only needs to be done once.

var _res_format_saver: MyResourceFormatSaver = MyResourceFormatSaver.new()
var _res_format_loader: MyResourceFormatLoader = MyResourceFormatLoader.new()

func _ready() -> void:
	ResourceSaver.add_resource_format_saver(_res_format_saver)
	ResourceLoader.add_resource_format_loader(_res_format_loader)


func _exit_tree() -> void:
	ResourceSaver.remove_resource_format_saver(_res_format_saver)
	ResourceLoader.remove_resource_format_loader(_res_format_loader)

How To Use

# Saving your data to the custom file
ResourceSaver.save(data, "path/to/file.custom_extension")

# Loading your data from the custom file
data = ResourceLoader.load("path/to/file.custom_extension")
3 Likes
16

@PotatoArmor This is neat.

This code would probably be better off in a plugin for the custom resource.

Gonna have to give this a go.

17

I have recently learned about this method. You can use it to handle data to and from a json file.

1 Like
18

From the Docs:

This method is performed implicitly for ResourceFormatSavers written in GDScript (see ResourceFormatSaver for more information).

1 Like
19

So while I got this working, I had a number of really weird issues with load order, and I decided it wasn’t worth a custom file type.

1 Like