Uploading Files via JSON

When creating the admin panel for Knížkomat, it was necessary to allow administrators to upload book covers, audio recordings and images in text. We decided to implement all three categories in the same way, i.e. a generic upload of a file to the server will return the address of the file and it will subsequently be saved where it is needed. However, during the implementation we ran into a problem with sending files through middleware that relied on JSON communication in both directions. So when we tried to send files from the frontend via the standard multipart-form, all sorts of errors popped up as the middleware tried to translate the request body to JSON. Of course, one would offer to make an exception in the middleware for file upload requests, but we decided, mostly just to deepen our knowledge, to work with this limitation.

File Conversion to JSON

We realized that a file, or more precisely the contents of a file, is nothing more than a long list of bytes. We are able to easily get this list in JavaScript in a format called ByteArray, and this can be serialized into JSON as an ordinary array of bytes. But this way we lose the information what type of file it is, it's not just the image/sound distinction, but also the exact type of image, like JPEG, PNG etc - the same goes for the audio track. That's why we added the extension of the original file to the JSON, so we can keep the correct format on the backend as well. It would have been tempting to send the full filename, but that wasn't necessary as the files are renamed on the backend.

Although this approach had no reason not to work, we were still surprised that the files were actually uploaded correctly to the server and did not break in any way during the transfer. At least that's what we thought. For relatively small files, this was indeed the case, but as soon as we crossed the magic 2.1GB threshold (which is not a big deal for audiobooks), we started losing the ends of these files. To be more precise, the magic limit was exactly 2,147,483,647 bytes, which corresponds exactly to the maximum size of the Java array in which the backend of Knížkomat is written. This seemingly random limitation comes from the fact that arrays in Java are indexed by integers, and their maximum possible value is 2^31^-1, or the aforementioned 2.1 billion bytes. Thus, once we tried to create an array of a larger size (more precisely, when the deserializer in SpringBoot tried to do so), it had no choice but to discard the non-indexable data, and with it, discard part of the file. For the record, the maximum size of an array in JavaScript is 2^32^-2, because it uses the same 32-bit indexing as Java, but does not count negative numbers.

Correct Solution

Of course, the problem of uploading files had only one truly correct solution, and that was to modify the middleware to work with formats other than JSON. However, thanks to our non-conformist solution, we ran into a practical example of field size limits, which we only figured out because, regardless of the size of the file being uploaded, the result was always at most 2.1 GB in size. However, transferring files in JSON can have its uses if we are sure that the files will not be too large and for whatever reason we do not have the option to use another transfer format.