Learning From Two Mistakes How Easily You Can Halt Production as a Software Developer

Wow. That is quite the mouthful of a title; a title appropriate for one who’s position is described between an intern and full-stack developer with less than two years under his belt.

First, some background context symbols which will be found throughout the article:

  • ${SD}: Senior Developer
  • ${UD}: UI Developer
  • ${BD}: Backend Developer
  • ${IFUG}: ME. *I Forked Up Guy*

Mistake #1: Return Promise<A, B, C, D>().then(DeliverDemCodes());

Even if you’re the best estimator the world had ever seen, learn as I never did that story ${A} will take an incredibly long time to develop due to stability issues. Acknowledge that ${D}, though perhaps an easy 2 hour task, may result in quite a bit of refactoring of ${C}. Perhaps you don’t have enough resources to even consider ${B} in the timeframe allocated.

In other words, in the world of bleeding-edged production and inexperience, never assume that development is a linear and constant variable.  Regardless of experience, knowledge, sleep deprivation; nothing is ever a ‘simple’ task which can be remedied in a defined time frame.  I was once delayed by two hours because my main workstation -with uncommitted code ready for attention,  decided 10AM would be the optimal time to update it’s security protocols and packages.

Mistake #1 Lesson:

${A} took 3 days, which is what the ${UD} and ${IFUG} expected. We did not expect ${C} to take a good chunk of 2 weeks, effectively causing us to leave ${D} on the storyboard. ${SD} was not mad, but also curious as to how I factored in the original estimation…. I told him I’m horrible at gambling.

Mistake #2: // TODO: Remove Legacy Code From Awesome Project!

You know how parent’s always respond to inquisitive questions with ‘because I said so’? I never enjoyed that as a kid, and thus is helped to form a overly-confident egocentric behavior that I wear at times; instead of asking ‘why’, or ‘should’, I simply do. I do, with zest and vigour. This can translate into questionable decision making skills at times, and even more questionable conclusions to basic logic.

Where am I going with this? Well one day I while developing around ${SD}’s code base with ${UD}, we had noticed a bit of UI work which was used only in a single, non-visible use case which had no application to the current scope. In the eyes of ${IFUG}, and the ${BD}s who were testing our work, it was an eyesore and issue to be tackled.  With precious time before the next presentation, one had to act brash and swiftly to remove the issue; not considering the potential risks / issues it would cause in the immediate future. My overconfidence that this was simply just a dead code from a legacy base was at an all time high, so I commented it out, recompiled, and committed the change.

Even if your IDE claims your code is unreachable / never used… don’t blindly comment it out and commit. When the tests pass, and the features aren’t breaking, don’t pass it off as a ‘fix’ and move on, neglecting to mention it.

Mistake #2 Lesson:

${SD} was not overly pleased with my actions the next day, for it turns out it was his configuration which was used on another branch of the same UI…. I had cost him hours of debugging only for his eyes to see the recent git diff, and discover that this genius right here (${IFUG}) commented out the code which configured his current feature. We had a good talk about that for the next few hours.

Conclusion

It’s very easy to get caught up in the development of a project, even more so when you’re given the keys to build the *arguably* most important layer. In the past few months, I’ve taken on more leadership roles & positions which resulted in my judgement also becoming a questionable state of mind at times. In that state, it’s very easy for flaws to burst through and expose themselves in your decision making skills. Ego, confidence, recklessness, these are the flaws which I had exposed in the later half of summer, and just a few of the lessons that I had to experience to truly understand.

Are ${SD} and I on good standing you may ask? Of course! Only after we established some rules which were settled on with fresh peaches.

A Semester Working Exclusively with MEAN Stacks

Since May, I’ve had the unique experience of working with MEAN stacks on a daily basis, each varying in complexity and architecture to reflect a different end goal. A semester ago, I’d never guessed how little time I’d be spending writing C++, Java, Swift, or even Python applications compared to JavaScript-powered web applications. Furthermore, this is the first time in my life that I’d been exposed to a technology stack not taught at Seneca, which during the time of my attendance examined LAMP, and C# / ASP.NET stacks.

 

What is a MEAN stack?

Each letter in MEAN stands for the technology platform used – similar to a LAMP stack (Linux, Apache, MySQL, PHP), MEAN stands for MongoDB, Express, Angular, and Node.

MongoDB – The persistence Layer

Express – The back-end layer

Angular – The front-end layer

Node    – The renderer layer

 

The Learning Experience

I explained in a different blogpost how little I knew of modern-day ES6+ JavaScript, and how easy it was to fall into a spiral of constant peril while trying to learn said new technologies. If it weren’t for David Humphrey’s brilliant instruction, I can imagine within a matter of hours I’d quickly become discouraged to the point of dropping the stack all together. Luckily, that was not the case.

 

MongoDB

Luckily for me, I’ve only had to learn the basics about MongoDB and how it relates to the data you see in your various mediums. It’s a fantastic NoSQL database tool which really helped me to learn the benefits and downsides to non-relational databases in a variety of contexts.

 

Having data saved as BSON (Binary JSON) is quite the freeing experience compared to the programmed constraints of SQL-centric databases. Being able to insert entire JSON objects, regardless of the document’s structure, allows for a much more scalable and flexible database in my opinion.

 

Granted, this depends entirely on the purpose of the database. Need data to remain constrained to preset rules, configurations, and relations? SQL. Need a place to store your marked up blog posts, or to save the comments of an article within the article structure itself? NoSQL!  You wouldn’t want to save one’s most important information for example in a database which doesn’t enforce any constrains natively (though, drivers / mappers such as Mongoose do alleviate this issue very well).

 

Express

Express was an interesting beast, full of new paradigms and JavaScript-centric programming habits.  Coming from a PHP / .NET background, the flexibility of Express allowed for rapid prototyping and scaling of applications.

 

In this technology, I also learned how to write proper REST API programs which would power the back-end in the cleanest (at the time) way possible. I’m certain GraphQL (a newer technology which is already taking web development by storm) will become the successor to the REST API back-ends, but for my needs I’m content with the knowledge accumulated on REST practices. My URL end-points have never looked better.

 

Angular 4

This semester was my first foray into Single Page Applications (SPAs), which have an internal routing mechanism allowing for a single page load to access most if not all of the views. You learn rather slowly just how powerful Angular can be from my experience, because many opinionated workflows and APIs are hidden behind  a seemingly unforgiving platform complexity. Once you learn the basics, such as routing, services, components, child-views, then you realize just how much can be achieved by surrendering one’s self to such a framework.

 

Angular 4 does have it’s limitations, and this goes back to a similar topic of ‘what is the end goal for this program’? For example, I made my life a living hell by choosing Angular for a project which really, didn’t receive any of the benefits Angular 4 could offer, simply because it was used out of ‘hype’ and not ‘logic’.

 

Would I recommend learning / using this for other novice web developers? Absolutely! Angular 4 is a hot topic among enterprise and startups alike, and equally valuable for web applications which revolve around a SPA architecture.

 

Conclusion & Thoughts

If I had to describe the experience in a single word, it would be ‘perplexing’; this is a different word than I would describe the technology stack itself, which would be ‘influential’. There are quite a few hurdles that one has to get through before seeing truly remarkable results, but once one looks back at all the programming paradigms relating to a JavaScript-centric stack that was implemented, I’m certain they’d be amazed.

 

Working with MEAN technologies for the vast majority of the summer has allowed me to learn quite a few bleeding-edge technologies such as Web-Sockets, Webpack, Web Components, and SPA front-end frameworks. These technologies, though niche to the software developer or desktop programmer, have paved the landscape of open standards which must be supported by browsers, and likewise how one approaches the concept of a modern web application. Open Source advocates such as Netflix have contributed tens of thousands of lines of revolutionary code, all relating to the modern web & it’s various uses to the end user. I truly am grateful that I could immerse myself in such a trend which is transforming the literal internet for everyone, and though communities and developers alike are segregated on the current state of the world wide web, I am forever content knowing what I had learned, and what I was able to accomplish.

Taking Boostnote for a Spin

Recently, I came across a review conducted by It’s FOSS which described a new open source project aimed at improving the experience of note taking for developers. Titled Boostnote, here’s what I gathered after making the software my default note taking application and snippet manager -replacing SimpleNote in the process.

Website & Installation

Downloading the software for your respective platform is an uncomplicated process, the website supplying the files & overview of different features. A feature which caught my attention was the cross-platform availability of the note taker, all in credit to the wonderful Electron platform which is powering the vast majority of desktop applications as of recent. What does this mean? It means that I could utilize Boostnote on Linux, Windows, & MacOS devices. For this review, I primarily utilized the application on the later two platforms -my Linux box being moved to Mississauga during the time of this article. Still, I’d like to believe that due to the nature of Electron applications, the experience is ideally cross platform too.

Upon downloading the respective installer for your platform, the installation automatically sets up the program in a predetermined directory -on Windows, MacOS’ installation process simply involved copying the application to your Applications folder.

Finally, on first launch of the application you’re asked to choose a dictionary which will contain the various notes you make. Google Drive & Dropbox support is coming in future releases according to a tooltip, but I found no performance penalty when I created a folder directly in Google Drive (through the desktop applications) which would store the JSON notes. This way, I could keep all the notes synchronized while I or the team behind Boostnote implemented a dedicated server for cross-platform file synchronization.

Themes

Initially, as all applications default to, the user is presented with the bright brilliance which dulls one’s appetite for night programming, the bane to the common programmer: a light theme. One of the advertised features of Boostnote was the inclusion of both a light and dark themed user interface. I quickly swapped the stark white aesthetic for a pleasing dark, which along with various syntax highlighting themes allows for quite the customization of the application. Because it’s open source, it’s nice to assume that you could contribute and improve the the functionality as you require.

One item that I’m looking into, is creating a dark syntax theme which blends seamlessly with the dark theme of Boostnote itself. So far, I’ve gotten close by using the icecoder theme. My ambition is to make a Nord-based syntax & interface theme which I’d love to contribute back to the project.

You can see from the issue tracker on Github, the project is far from perfect. Some inconsistencies, bugs, glitches even, dwell within the current release, whereas enhancements and improvements are also being discussed for future releases. Overall, I’m impressed with the overall polish and current implementation.

Functionality

Updates

With all the looks, charms and dark themed components, one has to wonder if it’s the eye candy is to distract from the implementation itself -or if the implementation itself perhaps is even more beautiful than said eye candy. Well, let’s talk about the implementation, features, and general feel of this application.

Opening the application, I was greeted with a small update dialog to the right, simply notifying me of an update downloading in the background -not obstructing or delaying my work with the application. It seems the update runs in the background entirely, meaning the next time you were to run the application the update would apply itself. At the time of writing this document, Boostnote was at version 0.8.8 with 0.8.9 being released within an hour or few according to the application’s Github page. It seems that as more developers discover and play around, they’re equally contributing back to the project and fixing bugs.

The Editor

Once getting to the editor, the meat of the application some would argue, you’re displayed what could be described as the closest friend you’ve ever known; a text editor. Perhaps some aren’t as fond as I am with the text editing region of a development platform, but if you’re going to spend hours, sometimes days of your entire week staring at a single item, I feel the importance starts to weigh in.

Thanks to the customization I had done previously, which if I may add, were simply for preference and did not improve the editor itself in any way outside of it’s aesthetics, I had an immediately usable, and friendly-at-night interface. How does it work? Well because it’s a simple snippet manager, a note taking platform, I didn’t expect anything beyond the realm of simple text entry and syntax highlighting. Is it a powerful editor? No. It doesn’t need to be, and I think that’s what makes it so inviting for note taking. The lack of “bling” also results in startup time which rivals that of Visual Studio Code -another Electron app, and is only out-shined by the startup-king-of-text-editors: Sublime Text.

Overall, I couldn’t ask for much more from a basic note taking application on the snippet side.

Markdown notes are an interesting case, which can be created by selecting “Markdown Note” instead of “Snippet” when creating a new document. Because of the way Boostnote approaches rendering of the Markdown notes, which is to render the document after the user has become inactive on the document, I was confused at first when my text editor would disappear, replaced with the rendered version while I took a five second break to consider how best to explain the code snippet. This isn’t an issue at all, simply just a different workflow than the standard preview in one window, source in other.

One inconsistencies which I’ve seen so far, is the markdown preview not following the syntax theme I had previously set, instead resolving to the default. It’s too early to throw darts at the application, so once version 1.0.0 is released I’ll take a look into outstanding issues & update this post.

Organization

Compared to Simplenote, which employed a simple tag system which would continually dump ALL of your notes on your screen, Boostnote takes on the traditional “Folders” setup which I appreciate beyond words. Having to tag a C++ snippet as #cpp used to make me cringe, so I only see good things so far in contrast.  Likewise the ability to include multiple “notes” in a single note (ala OneNote’s notebook style) is useful for those who like to keep bigger snippets separated & organized. ItsFOSS cited an annoyance of not being able to drag-drop notes between various dictionaries, but I did not find that overly frustrating since my workflow rarely if ever calls for such maintenance. Again, different workflows, different preferences.

Search

I found the current state of the search features to be a mixed bag, expecting the search to run through and return to my code snippets which contained a simple int keyword for example, I got no such results. That being said, if I included said keyword in the title, tags, or even description, then that document would be returned within the blink of an eye. Needless to say, we will have to wait and see before judging.

 

Conclusion

This application has been a great mix of flourish & disappointment, of wonder and oversight. What else could you expect from an application which hadn’t hit the 1.0 version number? I’m happily watching the development of this application, contributing even in the near future, so that when 1.0 does roll around, it’s improved upon issues I or other developers have encountered while using it.
The rough edges shouldn’t keep you away from giving the application a chance, and perhaps you’ll even come to enjoy it as much as I did while writing about it! To conclude, I’ll agree with ItsFOSS’ last remarks, citing that Boostnote is not for everyone -not every workflow is efficient on this platform, nor is the interface attractive to everyone. It’s a hit & miss application, but a great start to one at that.

 

Reviewing a Peer’s Optimization

A Code Review Summary for SPO600

For the final project of the Software Portability Course, the class was tasked with reviewing the code of a peer who’d set up a pull request for Chris’ GLIBC repository. For my code review, I decided my good friend John would be a worthy candidate for my review musings, his pull request being found here for those interested in viewing.

There were a few stylistic issues that I brought up, all of which a simple fix would remedy.

The Code

The Stylistic Issues In Question

Throughout the GLIBC, a common coding convention can be found throughout the familiar and obscure files. In such convention, is the inclusion of a space between the function name and arguments. John’s editor perhaps did not detect this, and instead replaced all of his code with the common function sans-space argument arrangement.

As you can see below, the issue is a minor one which is easily overlooked.

Coding Convention Issue #1: Correct

Coding Convention Issue #1: Incorrect

Another convention issue, was the rearranging of declaration syntax for variables and functions. I won’t deny, the GLIBC’s coding style is unfamiliar to someone who’s experience with C derives from few courses, and I did question why that style of C syntax was deemed essential for the time. Perhaps to reduce line length? This idea does make sense on the low-resolution terminals utilized in the days of old, but does look rather odd to the modern programmer.

Coding Convention Issue #2: Correct

Coding Convention Issue #2: Incorrect

Conclusion

John’s optimizations enable support for 64-bit processing, which is a big improvement to modern servers & programs alike. Having gone through his code modifications for the review, I did not see any issues regarding logic or operations, which in the end will always be the make-it / break-it factor. He did damn good work with the optimization, and with the stylistic change I’ve mentioned above, I think the upstream developers will accept his code contribution without hesitation.

 

Introducing Thimble’s Console V1.0

A OSD600 Contribution Overview

This post will be one of my last related to this semester, specifically to OSD600 which has seen the class learning quite a bit about Open Source web technologies; contributing to Mozilla’s Thimble in doing so. More on such topics can be found here and there. Though I’ve mentioned my contributions before, -even sometimes becoming the main focus of an article, I thought this post would show how the console works at the current time. As of this moment, it would appear that a good majority of the first release version is complete, with UX / UI being the last remaining items that are being taken care of by yours truly, or Luke who’s a fantastic graphic designer working for Mozilla.

Introduction

This console has been a feature request for educators and hobbyists, enabling them a cleaner method of instructing or developing JavaScript driven web pages with ease. Soon, their request will be accessible within Thimble directly without any hidden settings, or complicated setup.

I suppose, being honest, there’s not too much reason to be as excited or as proud as I am -but to that, I say that this has been quite the learning experience; full of new platforms and practices that I had never encountered before. I’m damn proud of what I was able to learn with the instructions of Dave, Luke, Gideon and various others this semester, and equally as proud to say that I contributed to Mozilla in a way which will help benefit users in new ways. With honesty out of the way, onto the feature set of V1!

Important Note, all user interface & interactions presented below are not finalized and may change before release.

Resizable Containers

Like every good text editor, all elements and windows should be resizable to accommodate a wide variety of screen sizes, user preferences, and workflows. Thimble has been no different, which meant that this console too, had to be resizable. This is handled by a resizing plugin found throughout the project, which has made all the difference it seems when it comes to customizability and accessibility in the various toolings.

Console.* Functions

What good would a console be if it only displayed your shortcomings? This console goes beyond that, allowing for one to utilize standard console functions such as log, warn, error, time, timeEnd, clear, and assert. Other functions perhaps will be added before V1.1 ;).

Console Functions

Error Handling

When engulfed in a spontaneous 10-hour coding session, it’s easy for the split of a finger to cause typos, syntax errors, and inconsistencies across variable references all of which, resulting in an error being produced. In the console, the error is displayed similar to the standard stacktrace found throughout common IDEs & debugger tools. Below, you’ll see the current implementation, which is still being fleshed out.

Error Handling

Toggable Context

When starting returning to a project, or starting fresh with a blank template, the console does not appear. Instead, you’re presented with a toggle found in the bottom right which displays the console. Likewise, for those unaware of the toggle, the console automatically appears when a console-related function is called; convenient I’d say. If the console is instead unwanted, closing it with the close button will disallow the console to reappear until the user reopens the device.

Getting to This Point

It’s amazing how, from the very start of a semester how far you can go down the wormhole, effectively specializing yourself in one of the vast code pools which make up Thimble. I would have never guessed from my first contribution that I’d be working on a console which interacts with the injected back-end code, overriding functions with replaced logic that caters to how the user would interact with the basic console. Likewise my peers, roommate even, have discovered a section of the code base that no one else in the class had. In my case, here is how I got to this point.

Contribution 1: Issue #1635

This bug made selecting a link on the user’s dropdown menu borderline impossible at times to register.  Luke’s description of the issue was:

The items in the dropdown menus only work if you click the text of the item, and not the entire length and height of the highlighted item when you hover.

Luke's Issue Picture

Issue

Pull Request

Contribution #1 Fix

Essentially, this was a CSS fix, which resulted in me adding a few attributes to the a links which would fill out the list item object. A slight detour, was that when I say CSS, I mean LESS, an extension of CSS which is compiled into standard stylesheets. Having used SASS before, LESS wasn’t overly alien in syntax.

Contribution 2 & 3: Issue #1675 (Back End)

This was more of an enhancement, which Luke described as:

When writing JS, using console.log() to check in on variable values is really handy. Opening dev tools while using thimble isn’t a good solution because the screen gets really crowded. Is there a way to intercept console.log() calls and display them in Thimble? Any other solutions, can we add support for a “thimble.log” method?
Console Mockup 1

Console Mockup 2

Issue

Pull Request

These two contributions oversaw the development of the back end, which would intercept and handle the console-related functions that would eventually route said function data into the user interface. When Dave first discussed with a all-so ignorant young developer (me!), I figured this would be cakewalk; simplistic. Within the first week, I probably inquired half a dozen times via email or the Issue page itself to figure out where to start. It was evident to everyone except myself, I was in over my head.

Turns out to a inexperienced young developer such as myself, that Thimble operated within multiple iframes, each passing data asynchronously through dedicated functions found in PostMessageTransport.js and the various extensions. This meant, that 1) I had begun working in the wrong code base already, and 2) was completely lost as to how one overrides window functions. Before this class, my knowledge of modern JavaScript was limited, fragile even.

After Dave got me on the right track, by the end of the pull request I had changed 90% of all the code that this excited developer was trying to contribute at least a dozen times. These changes were all warranted, such as eliminating repeating patterns which should become their own function, or stylistic changes which would enable the new code to sit among the rest without standing out.

Why did this span two contribution periods for the class? Well avid reader, because the second period was getting the system to communicate with each other, and the third being the optimizations, extensions and testing of said functions so that the fourth contribution would consist of the user interface alone without too much backend work. The final commit count before the merge was 25 commits, which interestly spanned 71 lines being added and 4 being removed.

Contribution 4: Issue #1675 (Front End)

The final stage of the console implementation was creating the interface, which I was recommended to port from a Brackets plugin. I took the later option, which would in turn endorse Dave’s lesson of ‘good programmers are lazy’; well said sir. Well said. As I started porting the extension, it dawned on me that quite a bit would be redone to make it accommodate Thimble’s interface which differed heavily in contrast to Brackets -it’s forked base. Furthermore, many of the ‘features’ had to be evaluated since they followed a workflow which did not accommodate the standard style that Thimble presented. You can see a before & after I took a crack at the interface below, with Luke’s input as well thrown into the mix. It’s not complete, but I think it’s a step in the right direction.

Before

Console Port Before

After

Conclusion

To conclude, as I said previously, I’ve learned a lot. My peers and I have learned from those with decades of experience in the open source world; those who have helped pave the direction of many open source projects. Likewise, the code review and exposure to different toolings had enabled me to understand how Thimble’s code base -a project built on top of a plethora of other open source projects, looks and interacts as if a single developer coded every line. Learning contribution techniques specifically relating to Mozilla’s practices has also helped future proof our skill sets -that is if you share a similar belief; Mozilla’s developments and standards being the ideal technological standard for open source developers. If you’ve made it this far, I hope you enjoyed the read and get a chance to try out the first implementation of the console in Thimble.

 

To Fail, is to Learn Without Safety Nets

An SPO600 Project Update & Admittance to Failure

Part 2

In my previous blog post, I dissected the first half of the SHA512.c implementation found in the GNU Standard C Library. The reason for such debauchery of our beloved cryptographic function? Because in attempts to optimize; I did the polar opposite. Coming to terms with such a fact is a difficult endeavour; analysing why it couldn’t be any more optimized being the only solution at the present time. So, I’ll continue from where I left off on!

Line 166: void __sha512_process_bytes (const void *buffer, size_t len, struct sha512_ctx *ctx)

This function is called from the the external sha512-crypt.c file, which is a correction to my previous assumption blog post which claimed this function being called in the previously analysed source code. Instead, I’ll keep analyzing and perhaps venture into other files if time permits in follow up posts. No promises on the later statement.

The first code block is only executed if the ctx->buflen’s value is not equal to 0. The comment above gives a hint as to why we have this conditional check before processing, which is “When we already have some bits in our internal buffer concatenate both inputs first”. The first code block’s logic is as follows:

  • Assign the value of ctx->buflen to a size_t variable titled “left_over”.
  • Create a size_t variable titled “add” which is the value assigned by a ternary operation 256 - left_over > len ? len : 256 - left_over.
  • Perform a memcpy on ctx’s buffer at the left_over array index, using arguments buffer and add as well.
  • If the above condition resolves to true, __sha512_process_block is called, and then memcpy is called again with multiple arguments from ctx.
  • After that condition, the buffer is assigned to a casted const char pointer sum of buffer and add. Finally add’s value is subtracted from len. Completes the code block.
  • Assign the sum of ctx’s buflen and add to ctx’s buflen. This is then tested against a condition seeing if ctx’s buflen is greater than 128.

The next code block process the complete blocks according to the comments, with a condition checking to see if the argument len is greater than or equal to 128. The next few lines determine how a function titled UNALIGNED_P(p) should be defined, dictated by the GNUC version.

Line 205 describes a while loop which processes the buffer variable in 128 bytes at a time  it appears. The last code block moves the remaining bytes into the internal buffer.

Analysis

Going back to my original statement in part 1, it appears that if the defined SWAP algorithm could be optimized the entire process would see a performance increase. Likewise, in this entire area of functions relating to cryptography,  it seems to me that memcpy could be the potential bottleneck; It is called six times throughout the file, often in loops. Another student is optimizing I believe, but I could be wrong.

Looking into the while loop above, making the buffer be processed in 256 bytes at a time may be possible, but the logic would have to change to accommodate the expanded byte range. Furthermore, because this is cryptography I’m too uncertain to experiment with such modification of logic.

Had I started sooner perhaps, delegated more time, or even just had more time to truly focus on the other items related to SHA512 cryptographic, perhaps I would have found a sane way to optimize the functions through inline assembly or a complete rewrite.

Tests & Ideas

Below are some code snippets that I came up with while testing theories and possible optimizations. Note that as you’ll see, most are minor optimizations in this case, meant to prove that the optimizations of this function lies solely with memcpy. As I go around reviewing this post, I see that the only optimizations possible were small ones which removed an addition operator. This code is highly optimized it seems.

Line 147:

Replace

With

Conclusion

A good majority of the lack of optimizations were my own fault, the primary factor being a horrible sense of taste when it comes to choosing functions to optimize. The later, being that perhaps I did not put enough effort into this project, prioritising other projects much more in comparison. My previous function for example, segfault, contained much more potential for optimization. The better question, was what good would such optimizations do? In the end, it’s all a learning experience, and I’m glad that I was given the chance.

 

A Conclusion to My First Semester Dedicated to Open Source Technologies

A Quick Overview as the semester draws to a close.

This semester, I dragged mind, body and code through five courses which strengthened, degraded, and tested the absolute limits of how little sleep one can get. Of the courses, two had a central focus on understanding, contributing and leveraging Open Source technologies on a multitude of platforms. These courses were also the main focus of many blog posts, and as we come to a close, I thought I’d reflect. Previously to OSD600 and SPO600, my dealings with OSS simply derived from learning and playing with various Linux distributions; never contributing or modifying code bases to meet my demands, or even bug fixing for that matter. I’ll expand on this later, but let me start by saying this, I was, am, and forever will be unrefined in the context of programming and technology. No matter how much I attempt to learn, or how far down the rabbit hole I plummet to fix an issue, I, like many, will always be a student. A student, excited for lessons.

Lessons Learnt

Throughout the semester, we’ve been exposed to technologies which in some cases, are close to the cutting edge of open source technologies; Node.JS, Heroku, Swift, Angular, Mozilla. It seems that for every workflow, testing environment, and language, there’s a plethora of platforms and tools which help to push the capabilities or development into the what some would have called futurism (or, platforms that developers could only dream about in whisper) just four years ago. For a developer invested in studies and learning new items, it seems that the waters (of content, tools, languages, etc.) are far deeper than they already appeared to be.

Previously, I had demonstrated an childish anguish towards JavaScript and its related technologies, often citing that programs built upon it such as Electron applications, were a hack to appease the end user who knew no better. Likewise, even with the hype which was Single-Page-Applications (SPA), dynamic loading of tools, and a wide package repository powered by NPM, I still advocated for older technologies which whispered frail outdated tunes and a comforting absence of JavaScript.

It was illogical to have such a bias towards a language which was so powerful, used literally everywhere, and was accessible. I’m incredibly grateful that, I was forced to work with JavaScript by contributing to Mozilla’s Thimble, for it along with Dave’s lessons granted me a second chance to evaluate the craze. I, was wrong.

JavaScript is an amazing language once you dip your feet into the waters deeper than your basic web creation 101 course, and see just how far the language has come. Perhaps this anguish had derived from a superiority complex developed from learning object-oriented languages -such as the likes of Java, C++, C#, Swift, as the end-all, and be-all for the modern day programmer. Regardless, I can proudly say that through this semester, I’ve grown to appreciate the capabilities that JavaScript has offered to modern development on the web, mobile, and even hardware at times. I’ve even recently started playing with a MEAN (MongoDB, Express, Angular, NodeJS) stack for an upcoming project. What a difference a semester can make.

Likewise, on the opposite end of the spectrum, software portability has become a budding topic among application developers and open source advocates alike. With the constant looming requirement for X86 applications to be ported over to AArch64, the platforms which programs and developers work under is in a state of intrigue. Porting software, libraries, tools even is not a simple task, nor is optimizing said port beyond the compiler’s capabilities. Throughout the semester, I was exposed to the different methodologies of ARM64 vs AMD64 processors; their machine-languages, operator syntax, processing techniques. Before such lessons, I had always questioned why mobile phones used non-AMD64 based processors, instead opting for the AArch64 which provided benefits that I could never fathom. Now, the light can be seen at the end of the tunnel in faint glimpses, waiting for you to reach out to it.

Questions & Thoughts

One question which I never got a chance to ask, was how would one say they wanted to ‘claim’ an issue to fix in Open Source projects? In thimble, the students would literally claim an issue and Dave would assign them it; Chris did the same with the GLIBC library functions. If I were not a student, how might I contribute so that two people were not working on the same item, thus duplicating the efforts?

Licensing. Just from basic research I’ve conducted in the past, I gather there are dozens upon dozens of open source licenses -each with unique requirements or specifications. I don’t believe this was a topic among any of the classes, perhaps because of the political undertones? Complexity?

How does one, who advocates and dedicates their life to that of Open Source, make a living? Specifically, since this was answered previously, how does a developer transition from being a hobbyist contributor for example -fixing bugs in one’s free time, to working on the project or even within the company itself? Is it to assume they’d be financially stable working on Open Source platforms?

On Open Source Itself

As I mentioned previously, my interactions with Open Source technologies derived primarily through Linux Distributions. It all started with Ubuntu 7.04 after Microsoft Vista absolutely fried my last remaining node of patience for the infamous operating system. After years of trial and error, I had probably tried almost every Ubuntu variant, later branching out to  modern distributions of Fedora, OpenSuse, Arch, and Debian. Aside from Linux itself, and the various software packages which made up the Desktop Environments that I favoured to use, I suppose my other interactions was finding Open Source alternatives to the software I previously used on Windows.

While in Seneca, I started to research and learn about web CSS frameworks such as Twitter’s Bootstrap or Zurb’s Foundation, which was my first introduction to Open Source platforms purposely aimed directly at Developers. That’s where it began to make sense. I realized that the Wikipedia definition of Open Source didn’t encapsulate the concept correctly, despite what many liked to believe: denoting software for which the original source code is made freely available and may be redistributed and modified.

Open Source is an idea, a culture; a vast culture full of subcultures, projects, and innovations which do change the world within the blink of a passing day. This culture I look forward to diving deeper into, and learning as much as I can through it. Even if I wasn’t so drawn towards the penguin, I do think that my interests in FOSS wouldn’t waiver. When I have had Window’s platforms on any machine, my goal was similar to that on my Linux workstations: Use Open Source whenever possible, and use it in hopes that you will contribute back someday.

Conclusion

When in the FOSS related classes, I often found we ran out of time due to long discussions and tutorials -which though normal for a class, I wish didn’t happen as often. I say that because, that class has been literally the funnest class I can think of in all my semesters here at Seneca College. It was also a first ‘proper’ introduction to Open Source for many, so the atmosphere of the class itself was always a unique fusion of knowledge, passion, and shyness. As I write this, my list for courses to take next is also being revamped entirely to include if possible more Open Source centric courses.

Between OSD600 and SPO600, I think you are given quite the extension of old and new, monolithic and modular approaches to different Open Source projects. Each project teaches a unique approach, some catering to that of a standard GNU library, whereas others catering to electron-based desktop applications or web applications. The theory that Chris provided through SPO600 is invaluable, but perhaps wasted on yours truly who doesn’t plan on writing optimized drivers for AArch64 installations anytime soon. Likewise, Dave’s lectures on developing, understanding large code bases, and interacting with the community helped to augment Chris’ lessons too.

I suppose to conclude, the one fact I can say about Open Source, within respects to programming itself, is that the playing field is always changing. As I claimed previously, I learned a new tool, architecture or coding style weekly, sometimes daily! These dynamic changes to the world of IT keeps those looking to stay relevant on their toes, and always a student.

To Fail, is to Learn Without Safety Nets

An SPO600 Project Update & Admittance to Failure

Part 1

Introduction

This series of posts includes not just my admittance to failure, unable to optimize a previously selected function, but also how I learned from said failure in regards to style, logic, and also why the algorithm was already at peak performance.  The catalyst to this downward spiral can be found in my previous blog post, which described my first dance with unnecessary optimizations to the segfault handler in the GLibC library.  My alternative function that I had chosen was the cryptography celebrity, SHA512, which I will discuss more in detail below. After each analytical paragraph, will be an analysis as to why the that section of the SHA512 implementation in the GNU Standard C Library is already well optimized beyond what my capabilities can contribute to.

SHA512 Analysis

For those who have a copy of the GLIBC on hand, or wish to view the code on a mirrored repository to follow along, the location of the SHA512 implementation is ~/crypt/sha512.c. The last recorded code change (that is, ignoring copyright or license updates), was five months ago, which simply adjusted the names of the internal functions to a new standard.

Line 34: #if __BYTE_ORDER == __LITTLE_ENDIAN

This is the first logical segment after the C-styled includes of the endian, stdlib, string, stdint, sys/types, and sha512 header files, and to understand the first condition, a definition is needed to for Little Endian vs Big Endian, and the overall concept of Endianness.

Endianness

Wikipedia’s entry on the topic is very thorough, so I’ve included it’s summary which explains Endianness well:

Endianness refers to the sequential order used to numerically interpret a range of bytes in computer memory as a larger, composed word value. It also describes the order of byte transmission over a digital link. Words may be represented in big-endian or little-endian format, with the term “end” denoting the front end or start of the word, a nomenclature potentially counterintuitive given the connotation of “finish” or “final portion” associated with “end” as a stand-alone term in everyday language.

Big Endian

This format stores the greatest value’s most significant bit in  the smallest possible address, with the following most significant byte preceding. The image to the side provided by Wikipedia describes how a 32-bit integer would be stored in Big Endian format, with an example from the University of Maryland providing an example using 4 bytes of data; each byte requiring 2 hexadecimal digits.

If given the following data: 90, AB, 12, CD, Big Endian would store the data in the following manner:

Address Value

Little Endian

This format follows the opposite of Big Endian, storing the least significant byte in the smallest address. Using the same example from the University of Maryland, which better explains the diagram provided by Wikipedia, you’d have the following allocation:

Address Value

Line 35: _LIBC

This code block is checking for LIBC, which is the compiled library. If it does exist, then we include the byteswap header file, and define SWAP(n) as bswap_64(n). If this we do not register the LIBC definition, then SWAP is defined using the following code sequence:

Since LIBC is included 90% of the time, I think this is for the edge-case where someone compiled the crypt utilities alone. The final two lines of this function are the edge case where _BYTE_ORDER is equivalent to __BIG_ENDIAN, in which case the SWAP is defined as  SWAP(n) (n).

Optimization Analysis

I do not see any optimizations possible here, which is due to the overall simplicity of the function. In consequence to understanding this code segment, I looked into the a great article which explains the syntax and concept behind bit masking. The bit masking allows for rearranging of the bytes themselves in a very efficient manner.

Line 131: void * __sha512_finish_ctx (struct sha512_ctx *ctx, void *resbuf)

This is the first function which contains logic, with the previous functions initializing array variables with constants and buffers. The function itself follows this logic, which I’ll explain why is already at peak performance code-wise after:

  1. Take into account unprocessed bytes by creating a uint64_t variable which contains ctx->buflen. It’s named bytes in this case.
  2. Create a size_t variable called pad, which will be used later.
  3. If the USE_TOTAL128, add ctx->total128 and the bytes variable created previously, if not, we add bytes above to the array in ctx called total. The last step, if the value we just added together in the array being smaller than the variable in step 1, is to increment the total array at TOTAL128_high by one.
  4. Here, we make pad equal to one of two conditions, based on bytes being greater or equivalent to 112. The first, is pad getting the difference of the bytes variable, and 240, the later being the difference of bytes and 112.
  5. Line’s 151 & 152 describe putting the 128-bit file length in *bits* at the end of the buffer, doing so using the defined SWAP function. The process looks like this:

  1. This step processes the last bytes, calling _sha512_process_block which will be explained in the next segment.
  2. Finally, a for loop iterates through an uint64_t array called resbuf, assigning the index to the value of the defined SWAP function with ctx’s H array at the same index.
  3. Return resbuf.

Optimization Analysis

A minuscule difference, is that currently the variable pad which is defined on line 136 is never used or initialized until line 147. The operations which take place in the 11 lines between the two are steps 3, which do not interact with the variable in anyway. Moving ‘pad’ to be declared before line 147 could increase compiler optimization potential, hypothetically allowing for the variable to be located horizontally in memory which would enable a lookup performance increase.

Seeing the importance of the predefined SWAP function, optimizing it (if it were possible) would make the greatest difference from my overall analysis so far. These ideas is a concept mind you, meaning that I would not bet anything of value on my thoughts or my contributions just yet. They’re rough, unrefined, ignorant.

Conclusion

Regardless, the overview is interesting, and also quite enlightening as to how cryptography in this context works, advanced bit management in C, and the coding conventions found throughout the entire Glibc. Though no segfault, no swan-song parody of a program’s best state, SHA512 is quite the hot topic as of recent with the technical blogs highlighting recent git collisions discoveries; Linus’ opinion on the matter, and how the industry itself is responding. Observing how SHA512 is implemented in C is not an item that I thought I’d be doing if ever in my programming career, but I’m damn pleased with the chance to do so. Stay tuned for my next segment, which will look over the final function in SHA512.c.

An Introduction to Heroku

An OSD600 Exercise

Heroku

This week, the class was introduced to Heroku, which is described as, “a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud”. It was a first step for many of us into PaaS concepts, along with interact with such a concept. Luckily, Heroku which is a paid service, can be used  freely to open source projects such as our tutorial. Below, I’ve included Dave’s instructions which guided us through the process, along with any thoughts I had along the way before concluding with a link to my “healthcheck” function, and my repository which houses all the code. Without further delay, let’s begin.

Process

Express Framework

The first item, was installing the Express web framework, which allows for flexible Node.js web applications which are both minimal, and robust. To install, we followed the standard command

, which added the express dependency to the project.

Writing The Server Code

Provided with the code below, the server would utilize Express’s routing for REST API calls. I found this routing to be easy to understand, mirroring how Laravel handles API routing. My only complaint with the code below, which I sure has an answer for that I have yet to discover, is the lack of organization. All the API routings are displayed in the single file, which in this case is easy to read, but what of bigger projects?

Running the server locally requires the following command:

. If all is successful, you can access the server through http://localhost:3000/, and test your previously implemented functions through ~/validate and ~/format. If working, the server will return a JSON response.

Deploying to Heroku

After creating an account on Heroku, the next step would involve installing the Heroku CLI -which is supported on Windows, MacOS, Debian / Ubuntu with native installers, and a standalone which I used on both of my Linux distributions (Fedora, Arch). Once installed, we’d login with

, providing the credentials created previously. The next step for this item would be creating the Procfile, which describes to Heroku which command to run when running the application.

Finally, we deploy to Heroku itself. This is done by first running

in your terminal, which provides you with a random name for the application which is also the application. Pushing to both Github and Heroku is simple, as the previous command added your application’s Heroku repository, and requires only this command after adding and committing your respective files:

.

Launching the API

To launch the application we just deployed to Heroku, we use the following command

, followed by

if you wish to visit your application’s domain in the browser. My REST API for this tutorial can be found at this link.

Conclusion

This was my first introduction to utilizing a PaaS both in an open source context, and a developers. Every week, It seems that I’m introduced into another universe, driven by technologies and platforms, names and languages; all of which I had never heard before. Last week I learned of TravisCI, and this week’s introduction to Heroku only expands upon an already interesting set of topics which are beyond the reach of the standard curriculum. Curiosity demands my exploration of Heroku, and perhaps for future endeavors into REST APIs will be powered by the platform. Only time will tell.

Giving Life to the Console in Thimble

An OSD600 Contribution Update

This short article will elaborate recent developments to the Thimble developer console that I’ve been implementing, with the previous progress post located here. Since then, the interface has been ported from a Bracket’s extension which will serve as the base for the Thimble version. Below, is the current styling and functionality, which is all up for debate and experimentation to further extend the features and usability for developers and educators alike.

 

Original Console State
Console Extension, Original State.

In the original state, a few items were scrutinized, leaning on the development cycle to remove, update, fix, or completely reinvent various aspects of the console as I’d work on implementing deeper functionality, usability or features requested. These items are being shifted out, replaced with more appropriate behaviour, which I’ll describe below.

The current implementation of the console takes data from the ConsoleManager, which one of the object managers I had implemented in my previous release which would be used for this exact reason. What this means, is that through all the clutter and warnings which originate from external files, scripts and errors, the user’s console statements stands alone on the console screen. If you wanted to see what is occurring in the outside world -the world outside your code specifically, then the developer console, inspector and various other tools are meant for such the task.

A new pull request has been created which this time, is an implied work in progress so that Luke can give criticism and opinions which pertain to the look and feel of the console. Already, Luke’s given some great idea’s including minimizing the console, instead displaying a message badge beside the toggle element for said element. The idea itself I find very useful, and has sparked ideas from yours truly, Dave, and Luke. Another idea, once which may be more useful for the user, is to display the console when a console function first occurs. That is, to have it pop up at the current position only when the User’s JavaScript code requires it, and then be closed by the user when they are done.

Minimized Console Idea #1
Minimized Console Idea #1

Once completed, I’m looking forward to explaining how it works, along with items I learnt over the course of this adventure in another article. I’m incredibly excited, not even for the ‘contribution to an open source project’ aspect of it, but because this console enables a plethora of tools to educators and developers alike inside Thimble. Is this the next big thing? No. It’s just a simple console window. I do think that, by allowing users to even have a dedicated surface for their console.logs, the opportunity to better educate new JavaScript developers while granting access to standard debugging tools -their variables; warning messages, data logs, timers, asserts, is invaluable.