1

There is a lot of buzz surrounding the effectiveness of ChatGPT for writing code. However, I have received several inquiries on Stack Overflow and its Spanish-speaking counterpart, ES.SO, where users have requested ChatGPT to either write code for them or fix a bug, but ChatGPT could not provide a satisfactory response.

The most recent question that I found saying that the OP asked ChatGPT to fix a bug is TypeError: Cannot read properties of undefined (reading 'tarea'). A new user posted this question to ES.SO yesterday. It is about a script bound to a Google Sheets spreadsheet. The script was created in Google Apps Script. The code includes server-side code, JavaScript and Google Apps Script services, and client-side code, HTML, and JavaScript embedded in multiple iframe layers, including google.script.run to call server-side functions.

Google Apps Script is a platform launched over ten years ago. What the OP is doing is a very common pattern, but it has typical beginner problems, like missing to include event.preventDefault() on the client-side form submission. I have already posted an answer suggestion to call this method.

As someone with experience in programming, I would advise those just starting to steer clear of using ChatGPT as a programming tool without the guidance of an expert. While it may seem like a convenient option, many nuances to programming require a certain level of expertise to navigate.

However, I understand that some individuals, like the original poster, may be curious about effectively utilizing ChatGPT for code writing and debugging. In this case, I would suggest seeking out resources such as tutorials, online forums, or even reaching out to experienced programmers for assistance. It's important to approach programming with a strong foundation and a willingness to learn as much as possible to avoid potential errors and setbacks in the future.

Do you agree with the advice given above, considering your expertise in Generative Artificial Intelligence and programming experience? Kindly explain your reasoning for your answer. Alternatively, if you disagree, please suggest a different guidance.

Resources

6
  • We can consider ChatGPT as a source among others: copy-pasting some code in a project may help, but may also be dangerous if the person doesn't understand what the code do. Examples: copy-pasting a fork bomb or a code that allow a security exploit. Commented Aug 8, 2023 at 22:41
  • 2
    I don't know what was the technology used by MDN, but experienced users were astonished that an AI generated wrong information. If a less experienced user uses these information blindly, they may waste time and/or introduce bugs. Commented Aug 9, 2023 at 17:00
  • If the learning process is taking place in a formal educational setting then the policies of the institution concerning use of AI need to be taken into account. Commented Aug 17, 2023 at 15:51
  • 1
    I don't know how useful you find my comment below. I feel very strongly about this topic. If you found my advice useful, perhaps translate it into Spanish and offer it to others.
    – mangr3n
    Commented Oct 13, 2023 at 11:57
  • @mangr3n I appreciate your answer (I already upvoted). I'm sorry I didn't add a specific comment about it yet. I want to read it thoroughly before marking it as the accepted answer. P.S. I am trying to remember why I didn't vote / comment on the previous answer. I will read it later.
    – Wicket
    Commented Oct 13, 2023 at 15:01

3 Answers 3

2

As a programmer who got his start in high school in the late 80s, and began professionally in the late 90s, I would recommend to junior coders that they use ChatGPT in a different way. Personally, I've used it to explain some details about some areas of ML process that I didn't understand, and I've used to dig into areas of Category Theory that weren't familiar to me. To do this I engage ChatGPT (4) in a conversation about the topic by asking it to explain specific terms or ideas. Often I find that it uses jargon from the discipline that I'm not familiar with, and so I usually continue the dialog to investigate the jargon terms until I start to understand them clearly. Then I test my understanding by offering my understanding back tot he LLM. If I still don't quite get it, I ask the LLM to give some concrete examples in order to get more clear. I've done this several times, and each time it has paid off.

These LLMs are not truth generators, they generate textual output, code or text, from an embedded gradient landscape. They are best used to explain the structure and relationships in information, the patterns embedded in human language. They can effectively reproduce high quality examples and even relatively decent code on various problems. But during the generation process, there is no "checking" component to attempt to falsify and correct error in the output. As a programmer, you cannot expect to eliminate that part of the creative process.
The LLM is also missing a means of checking and validating the situational awareness it develops in the context window. The only thing it has is what you provide in your prompting, up to the point that content falls outside of the context window, and through it you setup the conditions in the attention layer that determine which sets of gradients are participating in the textual generation. Just like you, it could be missing relevant information when it's generating output.

As an example, GPT-4 is horrible at composition of simple geometric elements into an SVG that represents things I articulate verbally. However, it can produce a simple clean svg arrow... Go figure. I could posit why, and I think I'd be correct, but that's an exercise for another day.

My advice

  • DO: Use an LLM as a critical capacity and a generative capacity within your own thinking/programming/testing flow.
  • DO NOT: Expect it to be the Wizard of Oz with all the answers to your questions.

But, what does that look like in practice?

Into Practice

Understand Yourself First

When you look inside at your own process of creative knowledge work (which programming is) you realize that you are engaged in an iterative process. You iterate two things, generation and evaluation. You must be. Finding solutions to problems is not about finding canonical simple answers. If the problems were simpler, we wouldn't need intelligence, and if they're too complex, even our intelligence can't help. The complexity we're dealing with in programming solutions never starts simple enough for canonical, algorithmic solutions.
However, we can break problems up into smaller and smaller problems until we do find algorithmically solvable problems. Then we compose those simple solutions into larger solutions that produce solution spaces for larger problems. If we do this well, we end up with working systems that address a class of problems effectively. Those systems become useful components in working ecosystems that productively contribute to human existence.

Understand Generative AI

LLMs have been trained on information that includes many, if not most, of the solutions that mankind has found to various problems in many domains. These include programming solutions in the codebases that tools like Copilot have been trained on.

If you present an LLM with a solved problem, it will show you the canonical solution, but if you present it with an unsolved, or uniquely contextual problem, it will have to attempt to compose a solution out of the patterns embedded in the code it has ingested. There's no guarantee that this will work on the first pass. And thus you must evaluate it. Generation -> Evalutation.

Practical Practices with Copilot

  1. Writing comments > Generating code Use Copilot to write comments in order to think about your code.
  • Write you comments first, as this provides context to Copilot.
  • Give starter text and let Copilot auto-complete your sentences.
  • Fix it's errors of thinking.
  • Move to ChatGPT or Claude for longer form discussions about a topic that you don't understand well.
  1. Generate Code
  2. Back to 1, write comments to document the code.
  3. Fix/rewrite code.
  4. Write a test (using steps 1 and 2)
  5. Iterate steps 3 and 4 until the test passes.

I would highly recommend that a junior programmer find and cultivate a relationship with a senior programmer to mentor you. Certain types of knowledge and awareness can only be built through real world programming experience, and a mentor with experience can help to point out those subtle and relevant concerns that an LLM can't properly assess. However, there's nothing like growing authentic experience by making you think and decompose, and diagnose your own errors, both in code and in thought, for this the calm, kindly demeanor and consistent availability of the LLM to collaboratively engage you in your IDE, in your code, with your problems, and your questions... Well that's a recipe for rapid growth, if you can trust the process.

Those are my thoughts, after spending the last 6 months using GPT-4, Claude and Copilot to work out problems from design, to theory, to practical implementation. Just because I'm senior, doesn't mean I don't have to break down problems from the very highest level system descriptions, down to simple bite-size pure functions that are < 10 lines of code.

0

If we are talking ChatGPT, i think some of the poor showing is going to related to the 3.5 model, and to get GPT-4 - which is definitely better, you have to pay the 20 bucks, which gets the chat interface but not access at the APi level 🤨. Anyway, ChatGPT is okay at helping flesh out requirements. Code, that would require some prompt engineering, zero shot in-out is isn't going anything past a single function imo. However if you have it critique itself - the quality will be better.

Hallucination is a b-ch, and i think those relatively new dont click the magnitude of hallucination, the models really believing what it is saying, and filling in the gaps with total garbage, so that Does work okay!!

Chat was Instruct tuned but not codex tuned if I remember correctly. I believe some have had success by just debugging the output with chatGPT by feeding the errors back in.

So if a noob cant even understand the code, (and then i expect the prompt wont be fantastic either), gpt-3.5 turbo is not going to fill that gap, i agree wth you, but nobody is trying to build prod ready even small system this way , are they? on the contra side though, if a noob grabs the code and continually debugs by passing the errors to Chat.....search isn't going to offer any better user experience.

so if there isn't an alternative - the interactive nature could have value, i would say if it has to be 3.5, zero trust LLM style, patience ...and rather think of it as a learning experience than a sprint to deliver. LLMs are a fact of life now, learning their ins and outs, even the hard way has value. Paper below is pretty recent and might be worth a read.

Demystifying GPT Self-Repair for Code Generation

0

ChatGPT is like having a person you can ask, and this person is quite knowledgable but isn’t really concentrating when responding and makes many mistakes. That can still be very useful if you’re clueless.

That’s it. Simply treat it as a person you can ask. Then you can get helped every time you get stuck. You can ask it how to troubleshoot different problems. It can help you find syntax errors and logical errors. And if you ask it to code something for you, this can be a quick way to go from nothing to something, but it does make mistakes and cannot see the big picture.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.