Constant 400 status code respose using the HTTPRequest

[Mikk]

Godot Version

4.2.2.stable

Question

I’m constantly getting the 400 HTTP status code response while trying to download the shared file from Dropbox using the HTTPRequest class. This link works in browser without any issues. I’ve tried to use Postman, disabled all of the request headers and found that the Host header is the only required one so added it manually in case if it is not calculated automatically, but it still does not work. What am I doing wrong?

Here is the code I use (should return the Hello world! text file):

func _ready():
	
	const url_part_1 = "https://www.dropbox.com/scl/fi/"
	const url_part_2 = "fc5rwkff82yvy66ktagv7/test.txt"
	const url_part_3 = "?rlkey=wqzs2qbbfazbgj3ttcaik9pq2&dl=1"
	const url = url_part_1 + url_part_2 + url_part_3
	print("URL is: %s" % url)
	
	var request = HTTPRequest.new()
	add_child(request)
	
	request.request(url, ["Host:www.dropbox.com"])
	var response = await request.request_completed
	
	var status_code = response[1]
	print("Status code: %s" % status_code)
	
	var body_string = response[3].get_string_from_utf8()
	print("Body as string: \n%s" % body_string)

[Mikk]

A bit of further investigation led to the potential deviations from the HTTP data format used under-the-hood of the HTTPRequest class. I found that Dropbox is strictly following the HTTP standatd, but seems like Godot is not. Can anyone help me with this theory? Is it possible to decrypt TLS traffic sent by Godot using Wireshark or similar software and compare it with requests sent from browser/Postman?

[Mikk]

With even further investigation I found that the HTTPRequest sends the # character when it is a part of URI and it leads to the 400 status code response from Dropbox. According to this RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax it should not be sent to server. However, I’m not sure that it should not be accepted by server, it can just ignore this part like Google and most of the modern web servers do… Anyway, that was the root cause of the issue and I can manually remove this part before following the 301 redirect.