Deep dive · DevOps for data science
A set of DevOps labs asked for more than a model. They asked for everything around one: R and Python working in the same document, an environment that can be carried to another machine, a repository that renders itself, and an API in front of the model. The first time that API answered, it said two words. Much of the time on this project went to what sits around the model: the environment, the repository, and getting it to publish.
The labs were about everything around a model rather than the model itself. I set up RStudio, created the project, and managed its packages. Quarto let R and Python share one document, with reticulate as the bridge between them. renv kept the R packages consistent so the project could move between environments, and the labs taught me to create and understand virtual environments. Docker was the piece meant to make the whole project run the same way anywhere.
Git held it together. Every change went from my machine to a GitHub repository, where GitHub Actions rendered the models and the Quarto documents automatically.
I worked with linear regression and decision trees in R and Python and applied them to real data such as Palmer Penguins. For classification I finished two models, a decision tree and a random forest. None of that is exotic, and the models were not where the time went.
The labs then built toward an API for model predictions. I wrote the model's Quarto file, started the Python session, ran the API, and opened it in a browser. It answered: Hello, Swayam.
What it showed
It was the first time something I had built myself was running and visible rather than sitting in a script, and it made the next problem obvious. A greeting is the simplest thing an API can return. Every step past it gets more complex and needs more customization.
Much of the time went here. Four problems, and what each one took to get past.
Early in the labs my laptop had to be reimaged, and all of my progress went with it. I restarted from zero, ran into problems the first pass never had, and worked back to where I had been. It was also the clearest argument for the labs' premise: an environment that is written down is one you can rebuild.
The message named an object, but the fix was a package. I ran the error through ChatGPT and learned that reticulate had to be installed in R; it gave me the command for it.
None of my files reached the repository and nothing showed in the Actions tab. I worked through git add, commit, pull, and push until the commands made sense, then found conflicts in the branch and resolved them. The fix that worked came from a classmate rather than a search: open the terminal in Git Bash instead of the Windows terminal.
The numpy install worked at first, then stopped when I needed it to push. I ran each error message through ChatGPT, tried every variation I had not tried before, restarted my sessions and the laptop, and asked my instructor. Hours later, the exact command I had used earlier worked.
The method that came out of it is plain: read the error, try a fix, test again. There was always more than one way to troubleshoot a problem, and the fix did not always come from the first place I looked.
I completed the API portion of the labs, and the API answered in the browser. Two things stayed hard: deploying an API effectively, endpoints included, and Dockerfiles. Both were in the labs, and both are where my understanding ran out.
What stayed is the workflow underneath: an environment you can write down, a repository that renders on its own, and the habit of working a technical problem through instead of around. Containers and API deployment, the two things this project left open, are what the AWS program I am in now at IDX Exchange works through on a production application.