The Flaws and Strengths of ChatGPT

In our last blog post, we “interviewed” AI, or ChatGPT for that matter, and we were quite impressed with its capabilities. But, we only scratched the surface on that one, and decided to get more in depth in our research this time around.

This blog post goes into more detail about ChatGPT and its use in the development process. How can software engineers benefit from using AI, and what are worst to best use-cases?

Quick recap of what we learned about ChatGPT previously

ChatGPT is an AI assistant trained to help with a variety of tasks, including answering questions and providing information on a wide range of topics.

One of the main advantages of ChatGPT is its ability to generate human-like responses to a wide range of inputs. This makes it ideal for applications such as chatbots, virtual assistants, and conversation agents. With its advanced language generation capabilities, ChatGPT can hold engaging and informative conversations with users on a variety of topics.

Another advantage of ChatGPT is its ability to learn and adapt to new contexts. Unlike some other NLP models, ChatGPT can retain information from previous conversations and use it to generate more relevant and personalized responses. This allows it to provide a more natural and intuitive user experience.

However, it can be noticed that ChatGPT is certainly not perfect.

One of the main limitations is that it requires a significant amount of computational power to run, which can make it difficult to deploy in some environments. In addition, because it is a machine-learning model, it is not always able to generate perfect responses and may produce nonsense or irrelevant output at times.

Another potential concern with ChatGPT is its potential impact on privacy. Because it can retain and use information from previous conversations, there are potential concerns about how it may be used to collect and store personal data. This is an important issue that developers and users of ChatGPT will need to consider carefully.

To show you what ChatGPT can do, and what its advantages and limitations are, we asked our programmers, and we will show you this with specific examples, by giving tasks directly to ChatGPT.

Cases

Generating full websites using ChatGPT

In the first case, we asked ChatGPT to print the complete code for a sample webshop application by using HTML, CSS, and JavaScript. As we already mentioned, ChatGPT cannot run the code itself, but he can provide examples and assist you in writing code in various programming languages.

In this case, the quality of his assistance will depend on the webshop’s specific requirements and complexity level. It can provide you with examples and help you write code in various languages and frameworks, such as HTML, CSS, JavaScript, and popular web development frameworks like React, Angular, and Vue.

It’s also important to note that building a complex webshop requires more than just writing code, it requires a good understanding of web development concepts and best practices, as well as experience with the technologies and tools used to create web applications.

Firstly, we tried testing it out using vague descriptions of what we needed — it ended up returning code that was not functional, so we tried being more specific and asked for two specific features:

• Adding and removing items from the cart
• Summing up the total value of products in the cart

Here’s the CodePen link for you to check out the code that we received. Note that the end result really doesn’t look good, even when ChatGPT was asked to make it “prettier” and added some basic CSS to it, but it is functional, to an extent. It doesn’t round up the total value of the cart, and the overall layout is quite odd, but if you combined the raw output with a human developer to sort things out, it’s easy to see how efficient it might be in certain cases.

Using Django and Bootstrap to problem-solve

In the next case, we used Django as an example, although any language or framework can be processed. It is essentially a framework based on Python, which is mainly used for the backend of websites, quite popular, you can upload an MVP version of the site relatively quickly.

We asked ChatGPT to create a login using Django and Bootstrap 5, and here is what it looked like:

It was quite impressive how ChatGPT explained the whole process, from the initial setup and install, to later coding out the page itself.

After it finished this task, we asked it to make an example of an API endpoint by using Django Rest Framework that returns all to-do tasks sorted in folders in JSON format.

Rewriting scripts to new languages

ChatGPT, as a language model, is able to understand and generate code in multiple programming languages and can assist with rewriting code from one language to another. However, its performance may vary depending on the complexity and specificity of the code and the languages involved. In general, ChatGPT has a good understanding of many programming concepts and is able to provide translations that maintain the overall structure and logic of the original code. However, there may be some nuances or specific requirements of the target language that ChatGPT is not aware of, which could result in errors or wrong translations. It is important to thoroughly test the new code to ensure that it behaves correctly and performs as expected.

In our second case, we wanted to check whether ChatGPT is able to translate program codes from one programming language to another, and to what extent it does so successfully.

We have asked ChatGPT for an example of a function that takes a number of hours as an argument and returns the equivalent number of days (with any remaining hours), also it should have been written in JavaScript.

The next step was to write an equivalent function to the previous one, but this time to write it in Python, that is, to rewrite the existing function in Python.

Here is what it looked like:

By checking the code in Python, the result was quite good, the error that appeared was in the definition of the name “tuple”, where ChatGPT gave us the name “Tuple” instead of the small first letter “t”, and that’s where the error appeared.

So, what do we conclude?

No matter how powerful it is, it must always be manually (that is, a human) validated if it is correct. It is important to test the code used by ChatGPT to make sure it works correctly and gives the right output in the target language. This may include comparing it to translations made by humans.

But it should also be emphasized that ChatGPT recognized some things that we did not ask for in the request. In this case, we asked it to convert hours to days and ChatGPT added a part in case we don’t have a round value.

Its use as an assistant to a developer

One of the best aspects of ChatGPT is its ability to assist and support in completing various tasks. It provides specific and detailed steps to help users understand how to complete a task and breaks them down into smaller parts, which is particularly useful for complex and theoretical concepts.

To show you that, we asked ChatGPT a question that sounded like this — “How to Dockerize a PHP app?”, and we got a really high-quality step-by-step guide on how to do it.

After this, we asked him to give us more details.

It’s quite obvious to see that it does charms in the function of an assistant. The way it lays out the steps necessary to solve issues, and then explains each step in greater detail if necessary, it sets the foundation for developers to tackle the planning phase of breaking down the logic of the problem at hand with ease, helping them to focus on the code itself earlier on.

Conclusion

In conclusion, ChatGPT is a powerful language model developed by OpenAI that has the ability to generate human-like text and understand natural language input. This makes it a valuable tool for a wide range of tasks.

One of the main advantages of ChatGPT is its ability to generate high-quality text quickly and efficiently, which can save time and effort.

However, ChatGPT also has limitations. It cannot always understand context and meaning, and its text generation can be limited by the quality of the data it was trained on. Additionally, it cannot perform tasks that require creativity, critical thinking and decision-making, human emotions, and human values.

ChatGPT needs humans to assist in various tasks, such as curating and cleaning training data, monitoring and improving the model’s performance, and interpreting the model’s output.

It’s important to note that ChatGPT is not designed to replace humans but to assist them in completing tasks more efficiently and effectively. However, it can be expected that in the future, technology will continue to advance and more tasks will be automated, but that does not mean that.

One thing to add, Stackoverflow has banned the addition of answers generated from ChatGPT, this is a temporary policy intended to slow down the influx of answers and other content created with ChatGPT. What the final policy will be regarding the use of this, and other similar tools is something that will need to be discussed with Stack Overflow staff.