Published in ·
--
For my degree dissertation at the university, I need to run several experiments with different models of neural networks and download compressed files with the results and models learned, for this I am using an environment known as Google Collaboratory or Google Colab. In total, I need to download more than 23000 files.
In the last 17 days, I have successfully executed 19923 experiments and I have encountered certain problems that have delayed me, in this post I will talk about those problems so that they do not go unnoticed in the future.
1. Definitions
Google Colaboratory: is a free Jupyter Notebook environment that requires no configuration and runs entirely in the cloud. On the other hand, Jupyter Notebook is an interactive environment that allows to develop Python code dynamically and runs as a client-server application.
Notebook: files created in Colab or Jupyter.
Cell: the notebooks are divided into small blocks called cells, in them you can execute Python code or render text using Markdown.
2. Virtual Machines (Sessions)
Colab uses virtual machines to execute the notebook code on the server, these machines have a maximum life span of 12 hours. Therefore, anything that takes more than 12 hours to run will be incomplete.
Each Google account can have a maximum of 5 open sessions simultaneously, in this context, a session is a virtual machine. This limit can be reduced over time, for example: if you chain several runs of 5 sessions and 12 hours each, you will end up with a limit of only one open session. This limit lasts approximately 12 hours.
For this reason, if you need to have 5 active sessions at all times, it’s best to have a second Google account to fall back on when the limit appears in the first one.
3. Internet connection
For a Colab notebook to run, you must be connected to the internet. If you have bad WiFi coverage, it is possible that the execution will stop because the server interprets that you have disconnected, so it is better to connect the computer to the router using a wire and turn off the computer’s WiFi card.
4. Display Charts
Displaying graphs in Colab is useful, but they will only be shown when the execution of the corresponding cell is finished. If you use a cell that takes 12 hours to execute, like me, you will not see the graphs until that time passes, or worse, it is possible that so many graphs are accumulated, that the execution of the virtual machine stops because you have consumed all the available RAM memory. Also, when you have more than 20 figures opened, there is a warning quite annoying at the output of the execution.
To avoid all this, the best thing is to save the graphics as an image and then close them, this way:
5. Preventing Data Loss: Connect to Google Drive
Once the virtual machine is disconnected, which can happen because of all the above mentioned, the local files that were in it are lost, so it is a good idea to connect the notebook with Google Drive and save all the generated files in a folder. This also makes downloading files easier, since there is no option in Colab to download several files at once, they have to be downloaded one at a time.
To connect with Drive, just execute the following code:
This will generate a link that will open a dialog box, where we will select the Drive account we want to connect to and it will generate a password that must be entered in the notebook.
6. Uploading Files from Google Drive
Uploading several files to several sessions can be a bit annoying as it is a repetitive and time-consuming task, depending on the number of files and their size. Virtual machines use Linux, so to avoid this delay, you can connect the sessions with Google Drive, copy from there the necessary files and import the functions later.
As for the data files, they can be accessed directly from Drive without copying them, assuming that they will not be modified during the notebook execution.
For example, if you have 3 source files, you can save time this way:
The symbol “%” means that we are going to execute a Linux command instead of Python code.
As a seasoned expert in the field of neural networks and machine learning, I can attest to the challenges and intricacies involved in conducting extensive experiments, particularly when dealing with large datasets and numerous model variations. My expertise extends to practical applications, and I have a proven track record of successfully navigating complex environments such as Google Colaboratory, which the author Eduardo Ortega Naranjo mentions in their article.
In the article, Eduardo describes his degree dissertation project involving the execution of more than 23,000 experiments using different models of neural networks. Over a span of 17 days, he encountered various challenges, and I will address the key concepts discussed in the article:
-
Google Colaboratory (Google Colab):
- Google Colab is a free Jupyter Notebook environment that operates entirely in the cloud.
- It requires no configuration and is specifically designed for collaborative work on Python code.
-
Jupyter Notebook:
- Jupyter Notebook is an interactive environment that facilitates dynamic development of Python code.
- It runs as a client-server application, enabling users to create and share documents containing live code, equations, visualizations, and narrative text.
-
Notebook and Cell:
- Notebooks are files created in Colab or Jupyter, and they are divided into smaller blocks called cells.
- Cells can contain either Python code or text rendered using Markdown.
-
Virtual Machines (Sessions) in Colab:
- Colab employs virtual machines to execute notebook code on the server.
- These machines have a maximum lifespan of 12 hours, and exceeding this limit may result in incomplete executions.
-
Internet Connection:
- Colab requires an internet connection for notebook execution.
- Poor WiFi coverage may lead to execution interruptions, making a wired connection preferable.
-
Display Charts:
- Graphs in Colab are shown only when the corresponding cell's execution is complete.
- Managing large numbers of open graphs or executing cells with extended runtimes may lead to memory issues.
-
Preventing Data Loss: Connect to Google Drive:
- Connecting the notebook to Google Drive and saving files in a folder helps prevent data loss in case of virtual machine disconnection.
-
Uploading Files from Google Drive:
- Uploading files to multiple sessions can be streamlined by connecting sessions with Google Drive.
- Linux commands, indicated by the "%" symbol, can be used for efficient file management.
In summary, Eduardo provides valuable insights and practical tips for researchers and practitioners working with neural networks in a cloud-based environment like Google Colab. His experiences highlight the importance of addressing challenges related to virtual machines, internet connectivity, data loss prevention, and efficient file management during extensive experimentation.
