sdc updates

2/17/19 - Using the fs module system, I was able to write a csv generator function that iterates up to 2 million and create the records. Anything above 2 million iterations will cause memory heap issues. It would be nice to use 1 loop and iterate 10 million times but I can’t get it to work despite changing the memory allocation to 12000 using a node script. The workaround was to use promises and writeFileSync after the first iteration, followed by multiple appendFileSync iterations to add to the csv file. The time it took to generate the 10 million records was 3 mins. It took an additional 11 mins to seed the local database for a total time of 14 mins. Super quick and much more efficient! The total size of the csv file was 4.3gb.. A savings of over 50% compared to 7.8gb using the previous seed script. I plan to seed the ec2 instance in the coming days. Bracing myself for issues..

2/19/19 - I seeded the ec2 database via import method by csv file and it took 41 mins. The average times were 0.41ms - 0.71ms for queries using primary key. If querying by a random attribute, the times were around 6-10ms which I’m happy with. I am going to deploy the app to ec2 in the coming days as well.. That way I can use the New Relic and Loader.io to see how the deployed app/database responds to the stress tests.

2/23/19 - I deployed my service to ec2 using Docker. Created a Docker image, pushed it to my DockerHub and ran the container in its own ec2 instance. The tricky part was understanding how to create the Dockerfile with the correct commands and pulling the image into the ec2 instance after installing Docker there as well. After that, it was just a matter of mapping the ports correctly so it would run. Now both my service and database are deployed in separate instances. I’ll stress test using Loader.io and New Relic in the coming days.

2/24/19 - Forget to mention, had to recreate the Docker image because Loader.io requires a token .txt file in your app in order to verify and give you access to its site testing. Took nearly half the day during class to get this to work.. Tried initially via the shell to move the .txt file into the ec2 instance but was not successful. The benefit of creating Docker images is having the portability of installing the service app to any remote computer… adding that file and recreating the image did not take long.