I merged 2 pull request that were co-authored with Claude Opus 5 - I prided myself on not having to disclose, but now I have to either disclose or reintroduce bugs

Help
1

Godot Version

4.4 and up

Questions

I am sad that I merged co-authored code. I verified the quality myself and I am quite convinced the contributor properly vetted the code as well.

My first question to more experienced developers: do you agree with me that the bug fixes in these 3 pull requests are indeed of good quality and an improvement or did I miss something?

Is my disclosure text any good?

10422
104221116×2484 312 KB

If you were someone considering to use a plugin that really solved your problem would you now decide to stop to consider using it?

Knowing that I am not as good at geometry and geometric algorithms as the contributor, knowing that if I were to handcraft this code I would still be looking at these fixes which are correct (effectively being co-authored on myself), knowing that I abhor reintroducing bugs that really don’t need to be there, knowing that Geometry is not patented afaik, would you advise me to purge the code and try to do it all from scratch?

Or shall I just leave the disclosure out there and accept my fate?

I abhor reintroducing bugs, but I’m seriously considering it… I simply can’t unsee the fixes, though. Catch 22 is what they call it, right? Checkmate..

2

Firstly, I think you maintain the code so it should be up to you whether you find the AI code maintainable. Since it seems you do, this sounds to me like it’s more about messaging around the AI PRs.

IMO you could label the AI-assisted PRs as such in Github and just have a blanket disclaimer sth like “Some code has been authored with AI assistance. These PRs are labeled and preserved for everyone’s review. See [link to filtered PRs].”

That’s transparent enough and allows anyone who wants to avoid slop to decide for themselves whether slop was added.

3

So you mean a filter like this right?

That’s a great suggestion, thank you.

4

I wrote this text up on the github:

I must admit, I have not felt this depressed about collaborating on code up to this day… I hope the joy will come back soon. It feels like the well was poisoned a little somehow.

It’s been humbling having to swallow my pride.

5

I’m sorry to hear you’re so down about it. I am also against generative AI in general, so I sympathize with the difficult choice this presented to you.

You’ve mentioned a couple of times that you didn’t know upfront that AI was involved in authoring the PR’s. Perhaps you could create some guidelines for contributing that might help protect you from that in the future.

For what it’s worth, these contributions won’t stop me from really digging in and learning your addon one day soon-ish.

6

Yes, I really should have. This just started as as a hobby that blew a bit out of proportion. I really was pleasantly surprised by the adoption so far and the collaboration. It’s been a bit beyond my expectation but still manageable.

I think I just need a small pause to get real life in order a bit. :blush: I promised myself it had to be fun, and keeping things fun requires a little effort too.

I really appreciate your responses and suggestions so far. Thank you!

7

It’s a slippery slope :smiley:

Your conundrum reminded me of this:

8

:rofl: Hahaha yeah… I may just be whacky guy now, though, the one that burns their own face :face_with_head_bandage:

9

Yes, I would. Avoiding AI slop is difficult because it infects core system components and not everybody does AI disclosure, but given the choice I will avoid AI output every time.

If you truly read and understand the PR, you should be able to implement your own fix for the problem(s) without copying the Claude output verbatim.

10

The human contributor and I myself both have reviewed the code line by line and the actual human made corrections themselves. It was co-authored, as they apparently call it.

The fixes include sensible automated tests and I tested many cases manually on top of those.

I’m confident about the code, but I don’t know whether I want to purge the codebase entirely to get rid of having to check this disclosure box.

I wouldn’t know how, frankly, because I have already been influenced by the chatbot’s suggestions.

In most cases rewriting would feel artificial: renaming a couple of ill chosen variable names like ‘loop’ .. I thought the contributions were entirely hand-crafted at first. Contributor could answer all my questions (it’s out there on the pull requests’ links I shared and in the issues, if you doubt my comprehension of the contributions)

It was only the speed at which they produced the code that amazed me, so I started asking these questions on the discord:

FIX #396: errors from editor by Clauveira · Pull Request #401 · Teaching-myself-Godot/ez-curved-lines-2d · GitHub - I just looked at the diff. It’s an impressive amount of code and I’m quite convinced it’s safe to merge.

A couple of questions, though:

  • Is it correct that you’d advise me to add free to the prune function? Do these nodes need to await being removed to do this safely?
  • How much time and effort have you put into this code?
  • Is it based on prior knowledge built up with experience?
  • Of course the plugin kind of boasts making curve updates at runtime easier via AnimationPlayer - do these extra deduplications and validations have a measurable extra performance impact or would it (in your opinion) remain in the same impact range?
  • And last one: if Geometry2D returns duplicate points in its boolean operations which impedes triangulation – should that not be reported as a bug - or at least an improvement proposal? We could co-author something (although it seems to me you wouldn’t need my help with that :D)

– I ask this because I usually learn by stacktrace (but godot does not always throw a great trace hint as to what caused my error) → did you chase the error by searching for it in the godot engine core, or is it simply the knowledge one builds up over years of game development?

  • also, you must have noticed in when your tests in console mode that my plugin leaks a couple of resource upon extit.. I suspect the arclist property, but I’m not sure… do you know?
  • how would I go about debugging? a c/c++ profiler?

You see, I thought it was all human. Because the code was vetted by a dev with 15 years of experience. And they knew what they were doing, it was solid enough code.

They asked the model a couple of these questions. I have the answers in a text file on my discord… Gives a nice insight into what they will have people buying Claude Opus 5 tokens for…

(Sorry for editing this answer so much, I will take a breather now, because I keep getting worked up about this whole situation)

11

@athousandships , @wchc . Is the checkbox on the asset store meant for unvetted vibe code and hallucinated art only? What do you make of my predicament?

I agree with @rainerdeyke that slop and unvetted code is a serious danger, but I know what I merged and it is not dangerous code. Is disclosure necessary in this case?

I hate having to check this box now. Maybe I’ll just branch off and ship a version with docile red error messages which only happen in edge cases with CollisionPolygon2D then.

12

I don’t know the rules for the Asset Store but AFAIK it’s for all generated content, regardless of quality or intentionality, if it has AI generated content, regardless if you added it, it has to have that checked I’d say

See the documentation:

Yes, however you must disclose what is AI generated, and how AI was used during the creation process when submitting the asset.

“AI” refers to any LLM/generative AI model like ChatGPT, Claude, Grok, etc., or any other models that fulfill the same role.

If you’re altering something AI generated, or using something AI generated inside of something created manually, that still counts and you must disclose it. For example, if you AI generate an image, and then tweak it manually afterwards. Or if you AI generate an image and put it inside a human created image, for example an AI painting in a drawing of a room.

13

Exactly. This is why I checked it

But if I were to rewrite the code by hand now, which I can, I would still be copying the logic, because some of it I understand but couldn’t have come up with myself. The Catch 22 I mentioned.

Will it be enough redaction then?

14

I’d say that doesn’t change anything, as long as the code is there regardless how rewritten it is it still is there, same as you can’t just take code from some closed source project and rewrite it and it not being a violation of copyright

15

So if you were in my shoes, what would you do?

Reintroduce the edge case bugs, knowing that when I fix them it will be influenced by ‘Claude’ somehow so I will just have to wait for the person with the maths major to help out? Trust them not to use a bot?

Or leave the checkbox on and hope most people will understand?

16

So my question here is:
Are you able to verify the correctness of the code? If you aren’t able to fix it yourself, can you trust the model fixed it correctly?

I don’t know what I’d do in that situation, it’d depend on my knowledge in the area and the complexity of the changes

If I couldn’t trust my ability to verify the changes, and couldn’t trust the fix was verified correctly by the author of the PR, I’d remove them, but I don’t know the exact situation

17

No, that would be a long read of this post, the pull requests and my discord in this stage.

Short story:
I trust and understand the fixes l. I could make them myself now as if someone had helped me on stack overflow ten years ago, but not from bare top of my head.

18

Then I’d say the options would be to keep them and disclose them, or rewrite them yourself from scratch, but if the code remains the disclosure should too I’d say, unless it’s rewritten from scratch

But keeping them wouldn’t be a problem otherwise, as long as the disclosure is clear

19

I would think of it like “clean room design” (term in reverse engineering). There are some interesting parallels

Now that you’ve seen the LLM-generated code, you can’t really unsee it. If the policy is that no LLM code is allowed, then the PR can’t really be used in any form without contaminating your codebase. Or you have to mark it.

I think it’s all totally absurd, I have no idea how to handle all the edge cases.

20

Note that there’s no rule to ban the code, but a requirement to disclose