Saving / Loading Resources in a Dictionary at 100k scale: Error Code 15 File Unrecognized

[dragonsdoom]

Godot Version

v4.2.1.stable.official [b09f793f5]

Question

I have a 26MB csv file that loads line by line to become values in a Dictionary -this results in a Dictionary of ~310k Resources. I estimate the size in RAM is probably like 200-400MB.

Loading this file every time at game start takes a bloody long time (~20 seconds) so I want to just serialize the dictionary, maybe compress it, then load the serialized dataset so it doesn’t take so long on subsequent loads.

Here’s my current boot code:

func _ready():
	if not FileAccess.file_exists("user://dictionary.save"):
		#do_the_first_file_parse()
		save_to_file("user://dictionary.save")

I’ve attempted saving the file a variety of different ways, using the different tutorials, but most of the simple solutions don’t seem to support the complexity of my resources. Right now when I run my save code (below), it creates a 0 KB dictionary.save file and then returns Error code 15 File Unrecognized. I don’t quite understand why this happens.

I know I was able to save with this filename in this directory using other Godot save code, I just wasn’t able to load effectively after creating the file using the earlier approaches. Any help would be appreciated!

Here’s the current save code:

func save_to_file(file_dir:String):
	var serial_dictionary:FileAccess = FileAccess.open(file_dir, FileAccess.WRITE)
	#save_game.store_var(valid_words)
	var valid_words_pack = PackedScene.new()
	valid_words_pack.pack($ValidWords)
	
	var result = ResourceSaver.save(valid_words_pack,file_dir)
	print(result, ": ", error_string(result)) #outputs 15: File unrecognized
	assert(result == OK) #code stops here
	serial_dictionary.close()

The packed scene consists of a single child node with this script:

extends Node
@export var valid_words:Dictionary = {}

Here’s the definition of a single Resource that would be instanced and saved in the above dictionary (310k of these):

class_name Word
extends Resource

@export var word : String :
	set(value):
		word = value.to_upper()
	get:
		return word
@export var def : Array[String] = [""]
@export var pos : Array[String] = [""]

func _init(new_word:String = "", new_pos:Array[String] = [""], new_def:Array[String] = [""]):
	self.word = new_word
	self.pos = new_pos
	self.def = new_def

func _print():
	print(self.word, ",", self.def, ",", self.pos)

func has_pos(pos_string:String):
	return pos_string in pos

func has_def(def_string:String):
	return def_string in def

func define():
	return self.word + ": " + self.pos[0] + " " + self.def[0]

[zeludtnecniv]

Hi,
You resource is simple enough to create your own save/load function. I made a test for saving using FileAcess function :

Save/Load Test

extends Control

@export var Generate_Count = 310000

var data = []
var loaded = []

func _on_load_pressed():

	var time = Time.get_ticks_msec() # time count

	loaded.clear()
	var file_dir = "user://save_game.dat"
	var file = FileAccess.open(file_dir, FileAccess.READ)
	
	# First load the dictionnary size
	var count = file.get_32()
	# Then load each word
	for i in range(count):
		loaded.append(Word.Load(file))

	print("Loaded in ", (Time.get_ticks_msec() - time) / 1000.0, "s") # time count
	
	



func _on_save_pressed(): # Saving function

	var time = Time.get_ticks_msec() # time count

	var file_dir = "user://save_game.dat"
	var serial_dictionary : FileAccess = FileAccess.open(file_dir, FileAccess.WRITE)

	# First store the dictionnary size
	serial_dictionary.store_32(Generate_Count)
	# Then store each word
	for w in data:
		w.Save(serial_dictionary);

	print("Saved in ", (Time.get_ticks_msec() - time) / 1000.0, "s") # time count





func _on_generate_pressed():
	data.clear()

	var time = Time.get_ticks_msec() # time count

	for i in range(Generate_Count):
		var w = Word.new("WORD", ["Pos 1", "Pos 2"], ["Def 1", "Def 2"])
		data.append(w)
	
	print("Generated in ", (Time.get_ticks_msec() - time) / 1000.0, "s") # time count

I modified a little your Word class :

Word

class_name Word
extends Resource

@export var word : String :
	set(value):
		word = value.to_upper()
	get:
		return word
@export var def : Array[String] = [""]
@export var pos : Array[String] = [""]

func _init(new_word:String = "", new_pos:Array[String] = [""], new_def:Array[String] = [""]):
	self.word = new_word
	self.pos = new_pos
	self.def = new_def

func _print():
	print(self.word, ",", self.def, ",", self.pos)

func has_pos(pos_string:String):
	return pos_string in pos

func has_def(def_string:String):
	return def_string in def

func define():
	return self.word + ": " + self.pos[0] + " " + self.def[0]


# Saving function
func Save(savefile : FileAccess):
	# Store the word
	savefile.store_pascal_string(self.word)

	# Store the pos size
	savefile.store_32(self.pos.size())
	# Store each string in self.pos array
	for t in self.pos:
		savefile.store_pascal_string(t)

	# Store the def size
	savefile.store_32(self.def.size())
	# Store each string in self.def array
	for t in self.def:
		savefile.store_pascal_string(t)
	

static func Load(savefile : FileAccess):
	# Load the word
	var w = savefile.get_pascal_string()

	# Load the pos size
	var s = savefile.get_32()
	var n_pos:Array[String] = []
	# Load each string
	for i in range(s):
		n_pos.append(savefile.get_pascal_string())

	# Load the def size
	s = savefile.get_32()
	var n_def:Array[String] = []
	# Load each string
	for i in range(s):
		n_def.append(savefile.get_pascal_string())

	# Recreate the resource
	var n_word = Word.new(w, n_pos, n_def)
	return n_word
	

Hope it’s clear enought

[dragonsdoom]

Thank you Zeludtnecniv, that solved my problem!

I am impressed! The filesize is modest, the load is fast.

In order to fully implement the solution I needed to make a few changes. In your example you used an Array, but I am using a Dictionary, so my code solution looks like this:

func _ready():
	if not FileAccess.file_exists("user://dictionary.save"):
		#do_the_first_file_parse()
		save_to_file_2("user://dictionary.save", valid_words.size())
	else:
		load_from_file_2("user://dictionary.save")

func load_from_file_2(file_dir:String):
	var time = Time.get_ticks_msec() # time count

	valid_words.clear()
	var file = FileAccess.open(file_dir, FileAccess.READ)
	
	# First load the dictionnary size
	var count = file.get_32()
	# Then load each word
	for i in range(count):
		var w = Word.Load(file) #notice temp var
		valid_words[w.word] = w #notice key-value assignment instead of .append() call

	print("Loaded in ", (Time.get_ticks_msec() - time) / 1000.0, "s") # time count

func save_to_file_2(file_dir:String, dict_size:int):
	var time = Time.get_ticks_msec() # time count
	var serial_dictionary : FileAccess = FileAccess.open(file_dir, FileAccess.WRITE)

	# First store the dictionnary size
	serial_dictionary.store_32(dict_size)
	# Then store each word
	for w in valid_words.values(): #notice .values() for Dictionary
		w.Save(serial_dictionary);

	print("Saved in ", (Time.get_ticks_msec() - time) / 1000.0, "s") # time count