
# Predicting Tornados
Training a 3D CNN on MIT's TorNet Dataset

[Github](https://github.com/meyersa/tornado-prediction)

## Contents 

# Intro

Finding an idea to work on for a thesis project is quite difficult. It turns out that it's something you spend a lot of time working on and even more time researching. Finding something that is intriguing but satisfies the broader goal is even more complicated. But then it hit me, I could be like Bill Harding from Twister and predict tornados, but use Machine Learning instead of a magical power. 

# Plan 

Just like any great project, it starts with realizing that MIT did it first (and probably better). 

Training machine learning models is all about having a quality dataset. Originally, I had been poking around NOAA's NSSL radar archives, but these are pretty intense. There are a lot of hoops to go through when downloading data and the amount of data they present is in much broader areas and is not labeled. 

MIT made this a nonissue with the [TorNet](https://news.mit.edu/2024/tornet-ai-dataset-carves-new-paths-tornado-detection-0429) dataset, which compiles radar images into a nice, easy-to-train dataset that has already been labeled. 

![TorNet sample Radar sweep](https://github.com/mit-ll/tornet/blob/main/tornet_image.png?raw=true)

# Implementation

While MIT had trained demo models on this dataset, they were limited by the size of their models, such as using a `2-D` Convoluted Network by dropping the timeline. 

Adding back this data timeline, we were able to make this into a `3-D` model, which transforms this into temporal data and has the opportunity to greatly increase the model's performance. 

## Procurement

MIT hosts this dataset on [Zenodo across multiple archives (2013)](https://doi.org/10.5281/zenodo.12636522). However, Zenodo is pretty slow for downloading, and planning for the future I decided to try and store this somewhere a bit faster. 

The first step was to download it locally with Aria2c, which took about 6 hours, and then upload it to Backblaze S3. 

```Python
def upload_files(file_list):
    """
 Upload files to Backblaze B2 using the b2 CLI.
 """
    for file in file_list:
 command = ["./venv/bin/b2", "file", "upload", BUCKET, file, file]
        try:
 logging.info(f"Uploading '{file}' to B2...")
 subprocess.run(command, check=True)
 logging.info(f"File '{file}' uploaded successfully.")
        except Exception as e:
 logging.error(f"Failed to upload '{file}': {e}")
            exit(1)
```

This was made pretty easy thanks to Backblaze's Python SDK. 

In the future to inhibit egress fees and better load across the world (important for later) it was put behind Cloudflare through a generic CNAME proxied record. For this, you simply add the Backblaze endpoint (f000.backblazeb2.com) as the content and then when you access it, it ends up being

> https://`proxy`/file/bucket/`file_name`

## Processing 

With the amount of files in TorNet, over 200,000, it just crashes the Python kernel on loading with Keras. To avoid this, the records are parsed with TensorFlow and serialized into a TensorFlowRecord (TFRecord) that can then be fed into the model. 

Without including too much code, 

```Python
with tf.io.TFRecordWriter(output_path) as writer:
    for file in tqdm(files, desc=f"Processing year {year}"):
 features, label = parse_nc_file(file)
        if features is not None:
 example = serialize_example(features, label)
 writer.write(example)
 local_label_counts[label] += 1  # Update local counts

print(f"Completed {year}: {len(files)} files")
return local_label_counts
```

The TFRecord can be called through a writer object which can write serialized data. Since this works through the TensorFlow data pipeline, it also scales across CPU cores making it a pretty efficient way to parse data. Notable, it does create very large records even with smaller data types that essentially double the on-device storage requirements. 

In this processing step, other things such as counting weight values, normalizing values, and removing NaN/flagging values occur. 

## Multi GPU 

Before training, it was also important to take advantage of multiple GPUs - since that's generally more cost-effective for training. TensorFlow provides a default strategy for this, again making it quite easy. 

```Python 
strategy = tf.distribute.MirroredStrategy()
```

## Parsing for Model 

With TFRecords created, they then have to be loaded back in and prefetched for the model. 

```Python 
parsed_example = tf.io.parse_single_example(example, feature_description)
```

Normally, data would have to be split here for training, but TorNet already had the dataset split - so they were loaded as two "separate" datasets. 

## Model Definition 

With the large amount of data, and crazy class imbalance (`189267` Non-Tornados to `13857` Tornados), I added a lot of complexities to sort of boost the abilities of training. 

- `Batch Normalization`: to try and aid in faster convergence and retain stable training. 
- `Leaky ReLU`: add additional activations to avoid dying neurons and enhance gradient flow. 
- `Spatial Dropout`: prevent overfitting by dropping feature maps randomly 
- `L2 Regularization`: add weight penalties to prevent overfitting 
- `Class Weights`: use the exact class amounts to avoid imbalance in training 

And of course, this is all done on a sequential 3-D CNN which had 6 3DConv layers and a final Sigmoid dense layer for binary classification.

All in, this ended up with about `886,337` parameters, which is not too many. 

## Training 

With about `200,000 images` and a batch size of 8, that led to about `25,000 steps`. As predicted, the training time with this model on TorNet was abysmal. On an RTX4060 locally, it was taking about `2 minutes` per step. Multiple that by 25,000 - and yeah, that was not happening. 

Moving to cloud GPU provider, [Vast.ai](https://vast.ai), I could then take advantage of bigger and faster GPUs, such as an NVIDIA LS40. Which dropped step time to `10 seconds`, and epoch time to almost `3 days` - which was still not going to happen. 

Finally, optimizing with multiGPU TensorFlow strategies, I landed on using 4 NVIDIA RTX4090s, which provided much better value for VRAM and TeraFlops, at about `$1.6 an hour`. With this instance, it was training at about `157 ms` per step, or about `30 minutes to an hour` per Epoch. 

> With Multiple GPUs, the batch was effectively multiplied by the amount of GPUs, so about 32. Meaning it required many less steps as well as training at less time per step. 

With a high training time, and a risk of the kernel still crashing, checkpoints were utilized to a directory that was synced to Backblaze. 

To ensure that only positive training would happen and stop the training as soon as it finished, EarlyStopping was added - and configured to default to the best weight. 

# Results 

In the end, the model performed okay - and even an improvement over other models that had been trained, such as MIT's 2-D CNN. 

![Result metrics](/images/projects/tornado-prediction/results.png)


Looking at Precision and Recall over time, it's clear the model was not fully converging, and it struggled to lock down on predictions. 

![Training vs. Validation Precision](/images/projects/tornado-prediction/training_vs_val_precision.png)
![Training vs. Validation Recall](/images/projects/tornado-prediction/training_vs_val_recall.png)

# Discussion 

In the future, I believe that the model could improve based on more context - which would help it converge better. For example, providing additional data from NOAA such as area weather information or Satellite imagery - in a combined model. 

This also provided a lot of insight into training as Vast's prices were not too bad, and are something I would use to aid in training time in other projects. Also as it turns out, it's nearly impossible to even get a GPU from other providers such as Oracle or Vultr, as they make you jump through tons of loops. 

Tornado Prediction


# Deer Identification
Combining Metadata with Images to test CNN capabilities

[Github](https://github.com/meyersa/deer-identification)

## Contents 

# Vision
In preparation for my Master's thesis, trying to better predict tornados by combining satellite images and Doppler radar, I decided to try a small project to grasp the combination of metadata for context with images. One of my professors offered up an idea for this, utilizing trail camera pictures of deer with the onboard sensor data. This data consists of the wind, the moon phase, temperature, time, and more, all things that could provide additional insight into deer behavior and predictions. 

# Goals
- Make a layered Convoluted Neural Network model for images and time series data
- See if Metadata improves model accuracy
- Try and beat the trail camera manufacturer's model

# Data Collection
The images for this project came from a trail camera, which uploaded images to the cloud. Gathering images would consist of inspecting the web requests, finding the endpoint, and finding the authorization token. Easy enough. After this, images would have to be downloaded from the endpoint, in this case, s3. 

At this point, I did come to the unfortunate realization that there were only about 3,500 images. This made the acquisition much easier but did not have great hopes for the model itself. 

Moving on to tagging, I utilized a small Python Tkinter program to pull up the image and then put the labeled result in a JSON. Not as impressive as something like RoboFlow, but it got the job done and allowed me to get through the 3,500 images pretty fast. 

![Labeling Program](/images/projects/deer-identification/label_program.png)

# Data Processing
Once images were imported, the metadata was cleaned up and dumped into a CSV along with the image location, which would be fed into the model. Additionally, the metadata would have to be regularized, and from suggestions of ChatGPT, formatted into standardes like cyclical and binary indicators. The images were then downsized to the usual 224x224, and the pixels were flattened. 

I had originally planned also to remove duplicate images - which became an entire process of trying to reverse engineer a Python library that was originally written to be a command line tool (which I got working) but ended up not being needed since it removed too many images and reduced the training set substantially. Also technically the images were not duplicates since the metadata would separate them, but that was beside the point. 

When processing the images, the small border at the bottom containing basic metadata was also stripped. It did occur to me that this strip was essentially feeding metadata into the model, rendering this technique a little useless, but the tornado data would not have this embedded so it still stood. 

# Training
With a mixed input of images and metadata, this model would be a little more complicated than things I had worked on before. My first idea was to simply train two models - and then combine them. 

The metadata would consist of a dense neural network, with only one layer since the information is pretty light. 

![Metadata-Only Results](/images/projects/deer-identification/Metadata-Only_Metrics.png)

While the image model would be a convoluted neural network, with a couple more layers. 

![Image-Only Results](/images/projects/deer-identification/Image-Only_Metrics.png)

To optimize this model with the small dataset, the amount of layers was kept small and dropout was not trimmed too aggressively. Additionally, I tried other methods like Transfer learning from ResNet or ImageNet but this resulted in lower accuracy. 

![Transfer Learning Results](/images/projects/deer-identification/Transfer_Metrics.png)

To expand the dataset, I also tried Augmentation, but that also lowered the accuracy. 

![Augmented Results](/images/projects/deer-identification/Augmented_Metrics.png)

And finally, the combined result of the image and metadata model. 

![Combined Results](/images/projects/deer-identification/Combined_Metrics.png)

# Results
As a baseline, the original trail cam manufacturer's accuracy versus my manual tagging was about `92.24%`. Unfortunately, none of my models came close to that - and unlike the manufacturer's model, they did not tag species or locations in the images. But this was to be expected as they most likely used some kind of YOLO model and trained on a lot more data - outside of the scope of this project. 

The really unfortunate part was that the accuracy of the combined model, at about `90.61%` was less than the image model, at `91.55%`. Looking at the low accuracy of the metadata-only model at `72.3%`, this is likely explained by inconsistent metadata with deer spottings. So maybe the idea wasn't the best... 

Until you consider the performance. The evaluation time for the combined model was `0.798 seconds`, compared to the image-only time of `2.058 seconds`. Showing that the improvements of the metadata substituting might not only show in accuracy but show significantly in prediction time. Over half the amount of time required. That's impressive. 

Looking at the accuracy graphs, the combined model does not do the worst either, in fact, it's pretty close to the best-performing image-only model. 

![Accuracy Comparison Graph](/images/projects/deer-identification/Comparison_of_Model_Accuracy.png)

And looking at the evaluation time graph, it also strays far away from the worst performing. 

![Evaluation Time Comparison Graph](/images/projects/deer-identification/Comparison_of_Model_Evaluation_Time.png)

Overall, I would consider this a success. While the accuracy took a hit, it remained high at `90.61%`, and seeing a noticeable performance increase over the image-only model is more impressive.

# Discussion
Many of the limitations faced by this project were because of the small dataset. The model had to be kept small, as complex models did worse by just memorizing specific things. Dropout and overfitting prevention were not an option as they would just forget all important details. In general, the testing data was probably just similar to what the training data was on and it probably just memorized features anyway. 

With a larger dataset, it would be interesting to return to this and try and gather more information about weather from other sources to make this a stronger metrics-aided image prediction model. 

If this worked well, it could be combined with other cameras to have a higher degree of accuracy in predicting deer in trail cameras and suggest things like where to hunt based on confirmed deer movement. 

Deer Identification


# CloudSCT
Disaggregating SmartCityTracker for the cloud

[Github](https://github.com/meyersa/CloudSCT)
[Live](https://cloudsct.meyersa.com)

## Contents 

# Intro/Inspo

To demonstrate the advantages of service-based architecture and cloud services in general - I took on a venture to modify an existing project into a cloud-ready project. In general, being cloud-ready means that the project is not based upon a single point of failure or monolithic service, instead, it breaks up as much as possible into smaller services. This provides advantages for performance being able to scale up the amount of services and being able to load balance across multiple servers. There are many more advantages, but that's something I'd have to focus on in an upcoming wrap-up for my migration to Kubernetes.

# SmartCityTracker

The perfect contender for this project would have to be something unique, interesting, and cloud-applicable. Originally, my goal was to use one of the previous senior design projects at CMU - however, it seems that none of them utilized GitHub so this was not possible...

In my search, I found [SmartCityTracker](https://github.com/caimeng2/SmartCityTracker), a database of smart city innovations that was scraped from Google and compiled into a viewable database. This project consisted of a [Zenodo](https://zenodo.org/records/7670784) database with all of the information and a [web UI](https://caimeng2.github.io/SmartCityTracker/) to display it.

The problem? The information displayed was downloaded from Zenodo as a JSON in a javascript file and uploaded manually into the index.html of the project. When you visit the site, all of this was loaded, despite only a fraction of it being displayed.

This means that every time the database is updated, the HTML root has to be uploaded, leading to things like gigantic nested loops of HTML.

![Nested HTML to load all of the data](/images/projects/cloudsct/nested-html.png)

# Disaggregating

To keep things fairly simple, the breakdown here would be to separate the frontend from the backend and the data from static assets.

## Frontend

The new service for the frontend simply serves static HTML when you visit the page which has been modified with API calls to load data when it is accessed - instead of all at once. When data is requested from the new API, it pulls it from Firestore, leaving it to be completely dynamic.

### Implementation

The first step for breaking up the datalink was to rip out all of the old. This meant the giant nesting would have to go, as well as most of the outdated Leaflet libraries - which restricted the kind of loading that could happen - or would have at least required me to find some outdated docs to do it.

The map works by displaying a layer of the Smart City data, in this case, it's a month and it's an entire JSON. To navigate the data, there is a slider, which corresponds to the month you are viewing.

![Slider](/images/projects/cloudsct/slider.png)

This complicates things a little as the slider requires all of the timestamps to be known for it to populate. Previously this was done by loading all of the layers at once into the slider, requiring all of the data. Breaking this up would be a little more complicated since it would have to be loaded with unknown data. Fortunately, Leaflet allows this to be done by just creating the input range (vanilla HTML) with the number of timestamps - which can be gotten with a Firestore query.

```JavaScript
const snapshot = await db.collection("data").get();
const files = snapshot.docs.map((doc) => doc.id);
```

And then fetched and passed through to the slider...

```JavaScript
sliderElement.type = "range";
sliderElement.min = 0;
sliderElement.max = timestamps.length - 1;
sliderElement.value = timestamps.length - 1;
sliderElement.step = 1;
```

Boom. Too much time was wasted trying to figure out how to do this with a Leaflet slider when the answer was so much more obvious. But, there was still the second half of this, dynamically loading the data into the layers using a GeoJSON? I still have no idea what that is - but that's the fun part.

Originally, this was done by loading all layers at once. And you cannot just add layers dynamically, instead as it turns out you can delete the current map layer and display a new one - which works just as seamlessly.

```JavaScript
if (currentGeoJSONLayer) {
    map.removeLayer(currentGeoJSONLayer);
}

// Add new GeoJSON layer
currentGeoJSONLayer = L.geoJSON(data, {
// Add icon
pointToLayer: function (feature, latlng) {
    return L.marker(latlng, { icon: customIcon });

},
// Add popup
onEachFeature: function (feature, layer) {
    layer.bindPopup(`
        <h3>${feature.properties.title}</h3>
        <p>${feature.properties.description}</p>
        <a href="${feature.properties.link}" target="_blank">Read more</a>
        `);
    },
}).addTo(map);
```

This solved some pretty major problems and made it easily off to the races. However, by far this was the biggest obstacle. 

### Closing thoughts

In a perfect world, the API could be separated into another service, but there simply is not enough traffic to justify this and the dataset is pretty lightweight as it is. However, if it came down to it, separately the two would allow for much easier scaling of the data. And if it came down to it, Firestore can directly be accessed by clients, eliminating the need for an API.

In a similarly perfect world, I would not have rewritten the entire project including the HTML and CSS, but here we are. Welcome to the fully redesigned Cloud SmartCityTracker!

![Web UI](/images/projects/cloudsct/ui.png)

## Backend 

With the data no longer being manually imported, it would have to be pulled through other means - directly from Zenodo. This fixes an entire step in the update pipeline and would make the project pull from a single source of truth instead of relying on multiple updates. 

### Pulling from Zenodo 

Zenodo is kind enough to offer a developer API, which allows for downloading files from projects - and even without API access. 

This meant that pulling the database would be as simple as... 

``` JavaScript 
const response = await axios.get(zenodoURL);
```

But the data is zipped which adds a fun extra step of unzipping. It was at this moment I also realized that Zenodo was not being used as a data source, instead it was shipped with the web UI. So on top of unzipping, the /assets/data directory would have to be pulled. 

```JavaScript
await new Promise((resolve, reject) => {
    fs.createReadStream(zipPath)
 .pipe(unzipper.Parse())
 .on("entry", (entry) => {
        if (entry.path.includes("/data/") && entry.type === "File") {
        // Copy to extraction dir
        const extractPath = path.join(extractDir, path.basename(entry.path));
        entry.pipe(fs.createWriteStream(extractPath));
        console.log(`Extracted ${entry.path}`);
 } else {
    // Skip
    entry.autodrain();

 }
 })
 .on("close", resolve)
 .on("error", reject);
});
```

With the datafiles located, it was time to upload to Firebase. 

```JavaScript
await db.collection("data").doc(documentKey).set(jsonData);
```

And just like that, the data upload pipeline had been finished. 

### Sike 

So as it turns out, the Github was the golden source of truth in this project and Zenodo has not been updated since 2023. Anyway, I'm still going to pull from Zenodo and pretend like this is the most recent data. 

## Firestore

This part is not as exciting and was actually quite boring. Basically, you just create a GCP Project to organize the environment, create a Firestore NoSQL database, create a service worker, add `Database User` to the service worker, generate an auth token, install the Node.js SDK, and you are off to the races. 

# Conclusion 

All in all, this project turned out to be pretty interesting. I love throwing myself into projects that I know absolutely nothing about and having to learn a ton of new information. In this case, I learned what a smart city was, how to interact with Firebase, and how to use Leaflet. 

I also reinforced my ego on web design. But that's not as important.

# Technologies

- Docker
- Node.js
- ExpressJS
- Axios
- GCP/Firebase/Firestore
- Zenodo


CloudSCT

 

# New Meyersa
Portfolio site reimaged

[Github](https://github.com/meyersa/new-web)
[Live (This website)](https://meyersa.com)

## Contents 

# Why does there need to be a V3
Simple. The current site, v2, was not up to par with my current skill level and the information was mostly outdated. Also, I was a little ashamed to include the link on my LinkedIn or Resume since it was a little ugly looking back... 

The iterations of portfolio sites began with [version 1](https://github.com/meyersa/meyersa-v1) to show off my experience and knowledge without just uploading my Resume PDF online. From the beginning, it was hosted on the domain [meyersa.com](https://meyersa.com) and on my own servers, which was pretty impressive for a freshman in college at the time. 

![v1 Website](/images/projects/new-meyersa/v1.png)

Over time this iteration of the site began to have more downfalls, making it apparent that a new site was needed. Since it was one of the first web projects I had ever worked on, it was not mobile-friendly and broke often on new displays. Don't get me wrong, it did not have any px values, it just was not very well thought out. Abstract ideas like blocks floating on the page just did not make enough sense to work without Javascript, and frankly, image centering and manipulation without Flex boxes made it quite atrocious. To combat this, a [v2](https://github.com/meyersa/meyersa-v2) was implemented with some more modern ideologies, but still without JavaScript to keep things simple. 

![v2 Website](/images/projects/new-meyersa/v2.png) 

The second iteration highlighted a new issue: being too basic. While it was more mobile-friendly, utilized better standards, and looked a little better - it still sucked. There was no pop, and honestly, it was distracting how blue it was. A more professional and thought-out website had to be built. Which is where [v3](https://github.com/meyersa/new-web) comes in. 

![v3 Website](/images/projects/new-meyersa/v3.png) 

# Vision 
The new iteration was going to come with some big changes to address most of the issues I had with the original site as well as some other goals I had in mind. 

## Making it More Complex 
While simple is usually better, vanilla HTML and CSS were not cutting it. This time it would be on React utilizing the NextJS framework. This would provide the customizability and reusability that I had loved working on [InstitutionPenguin](https://github.com/meyersa/institution-penguin). 

## Aggregating
At this point, there were two other sites, writeups.meyersa.com (defunct) and photography.meyersa.com (defunct) that were running. But they were equally too small to be their projects. Just like the portfolio on its own was too small. The new site would combine all three into one site that served a greater purpose. 

## Looks 
This time it would be mocked up before implementation and be planned ahead of time. Also catering to the photography portfolio, a viewer would be needed to ensure things looked better - a limitation of the previous WordPress site. 

# Implementation 
While there were a lot of moving parts with this, some of the main ones to consider are the React implementation, a custom photo viewer, and markdown parsing. Anything else would make this too long of a read. 

## React 
Instead of implementing React Native and considering a router to use with it, [NextJS](https://nextjs.org/) has had my attention for a while. I've used it in some other larger projects, and it works well when things get complicated. Under the hood, it's just a wrapper for React and it uses the same ideologies with some web-focused improvements. 

For example, NextJS allows for page templating, which can be used for dynamic entries like the projects directory on this site.

![Project Template](/images/projects/new-meyersa/project-template.png)

With the template, site content can be fetched from a local function, cached, and displayed to all visitors. Components can also be imported and used to display content - another great feature of React. In this instance, a component that I created, TitlePage, is rendering with information passed in from the Markdown files, such as the name and picture. In this component, I can then reference these inputs and use them to render content. 

![TitlePage Component](/images/projects/new-meyersa/titlepage-component.png)

There is nothing more satisfying than defining something once and then using it everywhere.


## Photo Viewer
On the Photography side of the website, images can be clicked on and viewed with more information in a less distracting setting.

![Image View](/images/projects/new-meyersa/image-view.png)

This was something I had always wanted to implement on the Wordpress implementation of my photography portfolio but never did due to the complexity of Wordpress plugins. With NextJS, things were different, it was an easy and approachable JavaScript task. 

The only challenge was popping an image out of its position on the page and making it interactable. Normal approaches would be to just absolute position it and overlay it on top - but in this case, overflow had been hidden to keep the images in a sliding display, making it impossible to pop out the original element. 

Luckily, there's another way: search parameters and react hooks. 

![Search Parameters](/images/projects/new-meyersa/search-parameters.png) 

By adding a query like the image number to the URI, a listener in a separate element can react by popping up the viewer. This eliminates any problems with overflow and even simplifies the process. Furthermore, any changes to the viewer like closing it and moving to another element, can be easily implemented by just changing the value in the query - something that would have been much more complicated otherwise. 

## Markdown Parser
To generate Dynamic content, the go-to solution is markdown. Instead of having a giant JSON that would be stupidly complicated to write every time or making the page for each article, markdown allows a way to just write a page normally and then parse it later into a web page. This takes care of all formatting like h1, p, images, etc., and leaves that to predefined styling - amazing. 

Parsing markdown with NextJS is pretty common, it's even featured in their documentation. However, as always, it's not that simple, and what I wanted to do was slightly different than what they wanted to do. Luckily, the frameworks that I used - remark and rehype, are platform agnostic and had their own documentation and I was able to figure out a way to parse just how I wanted - preserving images and even adding alt text to them later as seen in this post.

# Conclusion 
For the most part, this project has been wrapped up, but it never will really end. There are so many things I want to implement and there will be posts and styling changes coming with time. But importantly, now that it is on a real platform, these changes can be made without restarting the project and it will grow over time. 

# Technologies
- React/NextJS
- GitHub Actions CI/CD
- Docker
- JavaScript
- gray-matter
- remark
- swiper

New Meyersa

 

# CampusGrows.CMU 
Website to show awareness for sustainability/gardens on campus

[Github](https://github.com/meyersa/campusgrows)
[Live](https://campusgrows.com)

![What it ended up looking like](/images/projects/campusgrows/background.png)

## Contents 

# Intro/Inspo

One of my friends jokingly mentioned making a website for their RSO to raise awareness and I felt the need to take that seriously. While also having some fun. 

Another one of my hobbies is photography, so the idea of making a website with a lot of original assets sounded like a fun time. The result ended up being pretty nice. 

# Automation 

This project outlined a lot of cool parts (at least I think) of my infrastructure that made this more than possible. To start, having Github actions build a docker image automatically means that when I deploy (manually in this case) all I have to do is specify an image instead of building one. With the existing Traefik router on the main hosts, this also means that Cloudflare was pointing at the right place to begin with, making it possible to build this website and deploy it all within a single day. 

# Technologies 

- Nginx
- LR
- Photography 
- Docker

CampusGrows

 

# InstitutionPenguin
Senior design project involving agile programming, React native web display, API ready development, backend on NoSQL, and a full CI/CD pipeline with automated testing and deployment.

[Github](https://github.com/meyersa/institution-penguin)
[Live](https://institutionpenguin.com)

## Contents 

# Intro/Inspo

Tasked with creating a senior design project, we came up with the idea to make a web game that could feature some advanced technologies - such as being react native, CI/CD, API ready, and more. 

We also wanted to make this "educational" and decided to include some kind of topic we could base it off of. 

The final product was a web based game similar to Club Penguin, that talks about penguin habitat loss. Keeping it "educational", institution was added to the name for good measure. 

# Technologies 

- NodeJS
- NextJS
- MongoDB 
- Github/Actions 
- Cypress
- Docker
- NextAuth 

Institution Penguin

 

# AWS Flask: Small Eats 
A flask website used to showcase templating and dynamic deployment onto ec2 for ITC466

[Github](https://github.com/meyersa/aws-flask)
[Live](https://awsflask.meyersa.com)

![What it ended up looking like](/images/projects/small-eats/background.png)

## Contents 

# Intro/Inspo

One undergraduate class, ITC466 Cloud Automation, had a final project to deploy something on AWS using a couple of services. The obvious solution here was to deploy a website to EC2 and have it connect to RDS - knocking out two big services pretty easily. 

# Conclusion 

Despite all this work, somehow we were the only group who actually deployed something to AWS.. but oh well. 

# Technologies 

- Python
- Flask
- Jinja2
- Docker
- AWS 

Small Eats

 

# Making a NextJS Blog Site
Creating a spot to post projects using NextJS and other technologies

[Github](https://github.com/meyersa/writeups)

## Contents

# Intro/Inspo

Technical writing is something that is very common in Computer Science and Information Technology. Without it no one would really know what is happening, except the person who wrote it. For a while the things I hosted did not need any technical writing, because I was the one making it. Until it started to get to the point where I couldn't even remember what I had done. Then I knew it was time to make a blog of sorts. 

Instead of just spinning up another wordpress instance which I inevitably run into issues with styling, I decided it would be easier to make something more complicated. Introducing NextJS with remark to have the easy writing of Wordpress (kind of) in the style of markdown, but with the granularity of NextJS. 

# Process 

This can be broken down into a couple major phases 

## Research 

The first step was to research what technology to use. At this point I was in a sort of web phase, after working on a senior project [Institution Penguin](https://github.com/meyersa/institution-penguin). Our senior design project revolved heavily around a React router project called NextJS. It was pretty easy to use and it could do everything I wanted. 

With something like Wordpress the process of making a page is simply loading the site in your browser, logging in, and posting the page. However I felt like that was a little too easy, but at the same time making a whole page everytime I wanted to post also seemed too complicated. That's where Markdown came in. Using remark with NextJS I could make Markdown posts which would automatically get rendered into HTML pages. 

Now that I found the technology, the first step was to play around with it. The first generations were rough, basic code from other projects such as the NextJS documentation and other blog sites such as [Tina.io](https://tina.io/blog/simple-markdown-blog-nextjs/). This turned out to be rather useful, and I realized that this would not be too bad of a project to code. 

![Initial Code](/images/projects/nextjs-blog/initial-code.png)

## Implementation 

With some examples and guidelines found it was time to assemble a general NextJS site using my other projects as a reference point. This honestly took a lot longer than working with the JS for the posting since I had to dial everything into a point that I was happy with. 

> Working with the HTML/CSS of this project makes me consider another project, working on an asset hosting site that would allow me to standardize content between multiple websites with the ability to change CSS across multiple sites instantly. Sort of like an ansible but for websites. 

One of the first problems I ran into was that I didn't know NextJS as well as I figured. Combing through the documentation everything seemed familiar, until I tried to link to an external host for content (this time being github). While NextJS has great support for loading assets from the web, it turns out you have to explicity define that in the next.config.js file. Once that was figured out that problem was solved. 

Another problem I ran into was I had been working with Python so long that JavaScript syntax became a little confusing. Everything in Python just makes sense, its like writing English, but it's code. JSX with Next also did not help with this, it takes that syntax to even more complicated places. Especially when trying to pass arrays in to functions, which ends up being a mess of curly braces. 

## Styling 

At this point I almost feel like being a developer makes you into a part time UX designer. Websites never look good to me until I mess with them for a while. They start off with nice and neat CSS files broken up by module in Next, but end with random small fixes thrown in all over the place. (Part of the reason I want to get that asset host up is to remediate this problem). 

I really wanted to distinguish post locations, but using shadows or different colored boxes felt overdone after some of the other projects I worked on. My inspiration for this really came from the [Apache Kafka](https://kafka.apache.org) and [Apache Spark](https://spark.apache.org) websites. They look so clean, but are easily navigatable. It makes you appreciate the Apache foundations insane number of projects. 

Once again this process made me reflect on my other projects. Apache has so many websites, but they are not consistently themed. I have multiple, and it made me think about themeing them all the same, but maybe the color changes per site, but the rest is the same? Something about that sounds so satisfying. 

## Deployment 

NextJS provides some Dockerfiles that easily get the job done. Simply adding them and specifying the build directory took no time at all. 

And again I had Github workflows for deployment that I can reuse to have it get autobuilt on pushes. 

> Something I also want to consider is building images and pushing them to the Github container repo instead of building images locally. This would allow for more testing in something like a Jenkins pipeline before pushing, and also being able to then use that image in multiple places instead of having to build it in those places. 

And for now adding a route is easy, deploying with Cloudflare tunnels is just typing in the container name in the CF dashboard. In the future though, this will get moved into my Traefik mesh network on Oracle for better deployment standards and reliability.

## Making content 

With this blog site mostly done, it was time to consider content for it. I had unfortunately not really taken good notes of assembling this site, since it happened at random times and my memory isn't great, but now I had somewhere I could easily start writing a markdown file, or even just copy in from somewhere else, which should make the documentation process a lot easier. 

NextJS Writeups

 

# Shodan ETL Pipeline
My first attempt at an ETL pipeline using Shodan.io APIs to gather information about previous scrapes to build a dashboard on ExpressJS to display the information

[Github](https://github.com/meyersa/shodan-etl)
[Live](https://shodanetl.meyersa.com)

![What it ended up looking like](/images/projects/shodan-etl/background.png)

## Contents 

# Intro/Inspo

The inspiration for this project came from an assignment in one of my graduate classes, creating an ETL pipeline. ETL pipelines are common in data mining and other data science areas, basically extracting data, transforming, and then loading the data. 

Since I didn't want to just use a Kaggle dataset to operate this pipeline on, I chose instead to utilize Shodan.io and scrape information off there in batches and process it throughout the pipeline. 

> To continue with the ETL pipeline idea, I originally implemented it with Kafka connecting the steps. However, to make this more practical, the steps just send directly to eachother in a single container 

To further my web development experience, I templated this site with ExpressJS and I found it to be really intuitive and simple. 

# ETL 

## Extraction 

The first step, extraction, is taken care of with Shodan's own Python library. This just samples a query set in ENV and passes the information on to the next steps. 

## Transformation 

There are a couple steps inside of this step: flattening, cutting, and enriching. 

When the information originally comes in, it can be tons of different information per single record. To mitigate this, we flatten the incoming dictionary on IP and are able to save a lot of memory. 

To further knock this down, a lot of fields coming in may not be necessary. As such, they are dropped or concatenated when possible. 

Finally, incoming IPs are enriched with Data from Maxmind as well as Crowdsec to show more information about them. 

## Loading

Surprisingly, also another complicated step. We are regularly querying the same information and trying to store it, so a mechanism to only update the relevant information had to be implemented. Basically, if we have seen the IP before, we grab the current document in Mongo and compare it to the update. Generally, one piece of "data" changes and we can just modify the single field in the document. However, sometimes even the enrich data changes and in these instances it's easier to replace the whole document. 

# Technologies 

- ExpressJS
- Python
- Shodan.io 
- MongoDB
- Crowdsec 

Projects