Whether you are building an app or a game, Text-to-Speech can be very useful. For example, think of the different stages of game development - during concept and pre-production, text-to-speech can help build out the feel of the game and enhance your scripts before you record with real voice actors.

During release and production , it can be used to provide accessibility options to suit the needs of your users.  At the time of writing this article, there are over 456 voices across 147 languages that you can choose from!

As you are reading through this, you might be thinking to yourself...hang on...This uses a Cloud service, how would this work for an offline game? Or when a user loses connection?

This is where Hybrid (and Embedded) speech comes into play, and in this article, we are going to explore an example that will work both online and offline scenarios.

How Hybrid Speech works

Hybrid speech uses the cloud speech service by default and embedded speech as a fallback in case cloud connectivity is limited or slow.

In fact, you could ship your entire app with just the embedded speech and not use the cloud service at all. Its worth mentioning that it is slightly limited in that while the quality is good, the cloud option returns the highest quality speech. To give you and idea of what this looks like, lets compare the two versions. The first is the embedded speech version:

Embedded Text to Speech

0:00

/0:02

1×

And the the second is the cloud based version:

Cloud Text to Speech

0:00

/0:02

1×

If you listen really closely, you can hear that there is a slight improvement in the tone and cadence of the cloud speech service. There is not much difference, and the embedded version sounds pretty good too!

In this article, I am going to take you through a basic example of hybrid and embedded speech using Azure's Text to Speech Service.

Let's get started

Before we get started, we need to download the voices that we will use with the embedded version of the code. That is, the voices that will actually "ship" with the code. In order to acquire the voices, you will need to apply for access - follow this link to request access to the voices.

Once you have the voices, we can then start building out our example. First off, let's start by creating a new project in Visual Studio Code. Next, add a new class called Keys that will contain the keys and settings that we need.

public class Keys
{
    public static string EmbeddedSpeechSynthesisVoicePath = @"\voices\en-us"; 
    public static string EmbeddedSpeechSynthesisVoiceKey = "your_key"; 
    public static string EmbeddedSpeechSynthesisVoiceName = "en-US-JennyNeural"; 
    
    public static string CloudSpeechSubscriptionKey = "subscription_key"; 
    public static string CloudSpeechServiceRegion = "eastus"; 
    public static string SpeechRecognitionLocale = "en-US"; 
    public static string SpeechSynthesisLocale = "en-US"; 
}

Let's break down the code above. Firstly, the variable EmbeddedSpeechSynthesisVoicePath points to the file location where the voices are located and EmbeddedSpeechSynthesisVoiceKey is the key that you need to access the voices. You'll be given these when you apply for the access as mentioned above. I've also chosen the voice of "Jenny", but you could choose any from the Voice Gallery.

As we are using a hybrid model, we'll need to provide some cloud details from an Azure speech instance. I created a new speech instance on the Azure portal and on the overview page, I selected the CloudSpeechSubscriptionKey and CloudSpeechServiceRegion from the portal (highlighted in yellow below).

In the code above, we are also providing the instance region and the locale that we are going to be using in the variables SpeechRecognitionLocale and SpeechSynthesisLocale. In this case I'm using US English, but you could use any language and locale of your choice.

Configuring the Embedded Speech

Now that we have the keys configured, we can create the config for the embedded speech model. I've created a new class file called Settings and added the following code.

public class Settings
{

public static EmbeddedSpeechConfig CreateEmbeddedSpeechConfig()
{
    List<string> paths = new List<string>();

    var synthesisVoicePath = Keys.EmbeddedSpeechSynthesisVoicePath;
    if (!string.IsNullOrEmpty(synthesisVoicePath) && !synthesisVoicePath.Equals("YourEmbeddedSpeechSynthesisVoicePath"))
    {
        paths.Add(synthesisVoicePath);
    }

    // Make sure that there is a voice path defined above.
    if (paths.Count == 0)
    {
        Console.Error.WriteLine("## ERROR: No model path(s) specified.");
        return null;
    }

    var config = EmbeddedSpeechConfig.FromPaths(paths.ToArray());

    if (!string.IsNullOrEmpty(Keys.EmbeddedSpeechSynthesisVoiceName))
    {
        // Mandatory configuration for embedded speech synthesis.
        config.SetSpeechSynthesisVoice(Keys.EmbeddedSpeechSynthesisVoiceName, Keys.EmbeddedSpeechSynthesisVoiceKey);
        if (Keys.EmbeddedSpeechSynthesisVoiceName.Contains("Neural"))
        {
            // Embedded neural voices only support 24kHz sample rate.
config.SetSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Riff24Khz16BitMonoPcm);
        }
    }

    return config;
}

The code above uses the different file paths and keys that we set in the Keys class file. We'll be using this EmbeddedSpeechConfig object to create speech using the local voices on file.

Configuring the Hybrid Speech

In the same way that we set up the EmbeddedSpeechConfig object, we'll need to create a CreateHybridSpeechConfig object.

public static HybridSpeechConfig CreateHybridSpeechConfig()
{
    var cloudSpeechConfig = SpeechConfig.FromSubscription(Keys.CloudSpeechSubscriptionKey, Keys.CloudSpeechServiceRegion);

    cloudSpeechConfig.SpeechRecognitionLanguage = Keys.SpeechRecognitionLocale;
    cloudSpeechConfig.SpeechSynthesisLanguage = Keys.SpeechSynthesisLocale;

    var embeddedSpeechConfig = CreateEmbeddedSpeechConfig();

    var config = HybridSpeechConfig.FromConfigs(cloudSpeechConfig, embeddedSpeechConfig);

    return config;
}

You'll notice that the code above calls the CreateEmbeddedSpeechConfig() function that we created earlier. This is because we'll be using this code to use the cloud speech service by default and then fallback to the embedded speech in case cloud connectivity is limited or slow.

Try it out

With all this in place, we are now ready to start calling the API and synthesizing some speech.

/// <summary>
/// Synthesizes speech using the hybrid speech system and outputs it to the default speaker.
/// </summary>
private static async Task HybridSynthesisToSpeaker()
{
    var textToSpeak = "Hello, this is a test of the hybrid speech system.";

    var speechConfig = Settings.CreateHybridSpeechConfig();

    using var audioConfig = AudioConfig.FromDefaultSpeakerOutput();

    using var synthesizer = new SpeechSynthesizer(speechConfig, audioConfig);
    
    using var result = await synthesizer.SpeakTextAsync(textToSpeak);
}

When you call HybridSynthesisToSpeaker().Wait(), you should now hear something coming from your speaker!

In these code samples, I have created a simple string that we are passing through to the API. Depending on your use case, you could build more complex examples using Speech Synthesis Markup Language (SSML). Speech Synthesis Markup Language (SSML) is an XML-based markup language that you can use to fine-tune your text to speech output attributes such as pitch, pronunciation, speaking rate, volume, and more.

Using Embedded Speech only

If you preferred to use embedded speech only, you can call the CreateEmbeddedSpeechConfig() function that we created earlier.

/// <summary>
/// Synthesizes speech using the embedded speech system and outputs it to the default speaker.
/// </summary>
private static async Task EmbeddedSynthesisToSpeaker()
{
    var textToSpeak = "Hello, this is a test of the embedded speech system.";

    var speechConfig = Settings.CreateEmbeddedSpeechConfig();
    
    using var audioConfig = AudioConfig.FromDefaultSpeakerOutput();

    using var synthesizer = new SpeechSynthesizer(speechConfig, audioConfig);

    using var result = await synthesizer.SpeakTextAsync(textToSpeak);
}

When you call EmbeddedSynthesisToSpeaker().Wait(), you should now hear embedded speech coming from your speaker!

Summary

I've barely scratched the surface of the capabilities of text-to-speech; there is so much more to experiment with! If you'd like to learn more about Embedded/Hybrid Speech, I recommend reading the following article for more information.

In this article, we covered both embedded and hybrid speech options and its also worth mentioning that you can ship with only embedded speech if you prefer. In the code example that we ran through, we used C# but there are other language options available including C++, and Java SDKs.

If you had asked me at the start of working remotely whether I would prefer to work at home or in the office…..I would have chosen the office every time. As time has gone on, I have definitely started to lean towards working from home, but as you may know, that isn’t without its challenges either.

I’m always on the lookout to improve both myself and my team's daily work, so when I came across a book entitled Leading Virtual Teams by the Harvard Business Review, it really caught my eye.

Published in 2016, it  was written by the authors before the pandemic and before so many people started working from home. What I really like about this book is that the details written in here still stand, and it hasn’t changed the need for something like this. This book is part of the Harvard Business Review 20 Minute Manager Series, which means it is a concise, practical primer so you’ll have time to dive straight into the meat of the topic.

It starts off with some of the important things that you need to get right to be successful on a virtual team. First and foremost to this list is getting the right people on the team, which requires a special set of traits:

• Communication - “Good virtual team members know how to be precise and concise in multiple media, and the err on the side of overcommunication”.
• Collaboration style - “Virtual teamwork requires self discipline and self motivation, since team members must stay on schedule and ask for help when necessary. Remote work is not ideal for people who need a lot of supervision”.
• Temperament - “Look for people who will be generous in negotiating conflict in a low information environment and resilient working alone under pressure”.
• Technology - “Seek out people who are open-minded to new technology and competent in tools”.
• Size - “When it comes to the number of folks on your team, aim low. Research shows that smaller teams are more effective and more motivated”.

While you might not be in a position to build a team from scratch and choose people with these traits, it does help as a starting point to identify gaps in an existing team and tweak it accordingly.

The book also has a chapter entitled Manage the Technology in which it goes into detail about the importance of technology in the remote working world. I won’t go too much into detail here, as each organization has their own unique tools, but the key takeaway for me was around establishing rules for the use of the technology within your organization / team. How do you share and store content? What is the etiquette for the use of this technology? While the book doesn’t suggest which tools to use, it does talk about agreeing on using a standardised list of products. Regardless of whether you are going to use Slack or Teams - it suggests choosing one and sticking with it.

For me, one of the key areas of this book is around creating a shared vision with the team. What is the purpose of your team? Can you explain it in a clear, compelling language? Whether you are working with a team on a long-term project or a shorter goal, it’s important to clearly define what is the vision for the team and what will make it successful. By documenting this vision and goal, you can use it as a reference point to bring everyone back to common ground when distance and time chips away at the team's cohesion.

As a People Manager, I know how tough it is to keep a remote working team engaged and happy. The book goes into some detail and provides a few tips and tricks such as:

• Providing praise and recognising collaborative behaviour when you see it
• Encouraging people to acknowledge each other's work
• Playing games together!
• Finding a daily working rhythm with the team

Finally, Leading Virtual Teams also touches on the common problems that you might face with remote teams. There is a part of this chapter that is dedicated to managing conflict on a virtual team. It is not a question of if, but rather when you will face conflict with a team. Dealing with this remotely can be sensitive ground and will need to be handled differently to how you might do face to face. The book provides a few examples and ideas for how to navigate this territory.

Final Verdict

Would I buy this book? Yes, absolutely. It’s an easy read and is filled with practical, actionable details. There isn’t a lot of “fluff” - it gets to the point and leaves you with useful insights.

This book was written long before the majority of workers were forced to work remotely due to the pandemic. The authors have had real-world experience implementing this and faced challenges that they provide practical solutions to.

While some of the topics covered may come naturally to you and your organization, there are definitely some tips and tricks that you can pick up and use to improve your team's remote working. The book doesn’t go super deep into all of these topics and perhaps skims the surface a little more than I’d like, but after all this is entitled the 20 Minute Manager Series for a reason. I’d still recommend you buy this book!

When I have One on One meetings with my reports, I always take notes and jot them down on paper. The problem I found was that I never organised these notes properly and used them in the best way. I always had to shuffle back and forwards in my notepad and often lost notes. This gave me an idea and encouraged me to build a little something called Onesie.

Onesie is an tool to help you get the most out of your One one One meetings. It has features that allow you to take notes and help you improve engagement with your team.

Like most of us these days, I work almost exclusively remotely, and trying to maintain a connection with my team whilst being remote has been challenging. Without regular contact, One on One's can sometimes run of out energy and lose momentum.  For this reason, Onesie has a built-in function that suggests topics to talk about or important themes that you could run through with your employee.

Onesie is designed to work offline. So once you visit the site for the first time, it is cached and ready to use, even without a network connection. This can be particularly handy if you are on your mobile device and you lose connection, or perhaps working from a cafe and it has no Wifi.

Even if you aren't a people manager with line reports, Onesie is a useful tool to take notes from a conversation and then refer to them at a later stage (or the next time that you speak to that person).

Best of all, Onesie is completely free. I built this project as a tool for myself, but also out of love for the craft of people management. I would love for you to start using it, and welcome any feedback or improvements that you might have.

Head over to Onesie today to learn more.

Life often gets in the way of blogging, and I haven't made any new posts to this blog for a while. I've had a few on and off issues with Amazon EC2 Linux instances and this blog over time, but generally things were working as expected. The blog has largely remained untouched for a year or so. Which is why I was quite surprised to found out that I was greeted with an annoying HTTP 503 error on my site over the Christmas period.

I thought this might just be an issue with the site, so I tried the usual Stop Instance and Restart Instance. That didn't help.

After taking a closer look at the logs on Amazon, it turns out that I had actually run out of disk space on the instance. This is a bit weird considering I had 10 GB assigned to the volume - after all, this is only a small blog!

Increasing the size of the volume

My first thoughts were to get the site back up and running by increasing the size of the volume (or disk space) assigned to the instance. You can do this from the EC2 Management Console by selecting the instance, choosing Storage and the clicking on the Volume ID (highlighted in yellow below).

Next, select from the Actions drop down menu and choose Modify Volume.

Choose the new size of the volume and select OK.

Using Growpart

Once you've increased the size of the volume, it turns out that there is still one more step that needs to take place. You need to tell the partition to use the "new space" that you've just given it. Without doing this, it is still assigned the old volume size and you haven't actually made use of the new space you've given it.

You can see this by SSH'ing into the instance and typing lsblk in the terminal.

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   40G  0 disk
└─xvda1 202:1    0   10G  0 part /
loop1     7:1    0 97.9M  1 loop /snap/core/10444
loop3     7:3    0 97.9M  1 loop /snap/core/10577
loop4     7:4    0 55.4M  1 loop /snap/core18/1932
loop6     7:6    0 55.4M  1 loop /snap/core18/1944

In my case the partition xvda1 is still assigned 10 GB, but the root xvda uses 40 GB.

The simple solution is to run the following command on the root partition and tell it to start using the new space it has been allocated:

$ sudo growpart /dev/xvda 1

However, when I did this I was presented with the following error:

mkdir: cannot create directory ‘/tmp/growpart.2626’: No space left on device

Arrrgh! This meant that I didn't even have enough disk space to expand the disk space. I was screwed! One of the suggestions that was mentioned online was to use the $ apt-get autoremove command to remove those dependencies that were installed with applications and are no longer used by anything else on the system. Unfortunately, I was also met with the "No space left on device" error when I ran the command.

I searched for other files to remove, but being a bit of an amateur with Linux, I decided to delete the safest files....the log files. I did this by typing the following command in the terminal:

$ find /var/log -type f -delete

Whew! This bought me an additional 300 Mb which was just enough to run the growpart command again.

$ sudo growpart /dev/xvda 1

Success! If I run the lsblk command to verify that partition 1 is expanded to 40 GB, I see the following:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   40G  0 disk
└─xvda1 202:1    0   40G  0 part /
loop1     7:1    0 97.9M  1 loop /snap/core/10444
loop3     7:3    0 97.9M  1 loop /snap/core/10577
loop4     7:4    0 55.4M  1 loop /snap/core18/1932
loop6     7:6    0 55.4M  1 loop /snap/core18/1944

I ran the resize2fs command to resize my file system. It can be used to enlarge or shrink an unmounted file system located on device.

$ sudo resize2fs /dev/xvda1

Finally, I started the Ghost CMS again with the following command:

sudo /opt/bitnami/ctlscript.sh start

Once I ran that command I was greeted with:

ℹ Checking if logged in user is directory owner [skipped]
✔ Checking current folder permissions
✔ Validating config
✔ Checking memory availability
✔ Starting Ghost
You can access your publication at http://xx.xxx.xx.x:80
Your admin interface is located at http://xx.xxx.xx.x:80/ghost/

I never thought I'd be so happy to see the output from a terminal!

Summary

I still haven't been able to get to the bottom of why a small Ghost CMS should be taking up such a large amount of disk space, but for the moment I am happy that my site is up and running. I have Cloudwatch alarms in place to alert me if the disk space grows too large now.

If anyone has any guesses as to why this happens with Ghost hosted on Linux, let me know!

I’ve just finished reading the book and I can say that I definitely enjoyed it.

Through four years of research, the authors set out to find a way to measure software delivery performance―and what drives it―using rigorous statistical methods. This book presents both the findings and the science behind that research, making the information accessible for readers to apply in their own organizations. Personally, I think it's quite cool to see scientific data applied to some of the everyday things that we do in technology organizations. I also found it interesting to see how some of the top performing companies repeatedly seemed to pull away from the “pack” by simply applying many of the techniques applied in this book. The book takes a look at well-known best practices such as Continuous Delivery, Lean Management, and Transformational Leadership.

If you are looking for a book that tells you how to build and scale a high performing technology organisation, then this book doesn’t quite cover it. This book goes more into why you should be doing things a certain way and backs this up with facts.

Part 1 of this book explores what the research team found after trawling through the data. Part 2 dives into the science behind the book and an introduction to Psychometrics. Finally Part 3, looks into transformation, which focuses on how leadership and management can help drive these improvements.

If I had one takeaway from this book (which surprised me), it was that the leadership of an organisation plays a pivotal role in the team's results. I really liked this quote:

“Leadership really does have a powerful impact on results. A good leader affects a team’s ability to deliver code, architect good systems, and apply Lean principles to how the team manages its work and develops products. All of these have a measurable impact on satisfaction, efficiency, and the ability to achieve organisational goals”.

If you’d like a bit of insight into the book, I’d recommend checking out the following resource, which contains a useful table of high-performance team, management and leadership behaviours and practices.

Overall, this is a great book and definitely gives a good insight into why some of the best practices that we take for granted are worth doing and lead to high performing technology organisations. This book concentrates more on the why as opposed to the how. If you are looking for a how to, I’d recommend reading The DevOps Handbook to learn more.

I hope you enjoy the book!

Behind the scenes, I have been working on a new book with my EA colleague Jo Root. I am super excited to announce that The People Manager: A Guide for the First Time Manager has been published! This is a personal project that has been a year in the making.

The book is designed to de-mystify the challenges of People Management and give best practices, personal hints and tips from our own experience, and splitting up what might seem an overwhelming new role into clear, focused and hopefully manageable chapters of advice and coaching.

Becoming a People Manager is an exciting new challenge in your career but can also feel overwhelming. Being new to the role will no doubt mean that you have many questions that you want to find answers for. Your previous work experience, whilst invaluable for your personal development, may not give you the tools or confidence you need to springboard straight into your new role.

This book takes you through practical steps and real-world examples to become a confident, and ultimately successful People Manager.

Our hope is that you will:

• Discover tried-and-tested examples of tools and process you can use in your new role.
• Explore the role of a People Manager and learn about what to expect.
• Learn how to run effective and engaging one-on-one sessions with your team.
• Master the art of building happy teams.
• Learn how to handle conflict amongst your teams.
• Discover how to continually learn and grow in your new role.

We hope that new People Managers out there (or those returning to the role) will find this book invaluable for your learning, growth and success in your career.

The book is now live on Amazon and available to order!

When it comes to hiring for a technical role, more often than not, the emphasis might be placed on the technical skills of the candidate. Of course, if it is a technical role, it is vital to assess these skills. However, one part of an interview process that I feel is often overlooked is the soft skills.

At this point you might be wondering - what are soft skills? If you look up the definition of soft skills in the dictionary, it is defined as “personal attributes that enable someone to interact effectively and harmoniously with other people” and I think that it sums this up perfectly. These “soft skills” are also referred to as emotional intelligence and are an important part of the hiring equation. As a manager, soft skills are those “fluffier” attributes that are hard to define. Employees who possess these attributes are generally nice people to work with. I like to think that soft skills fall under the umbrella of creativity, listening skills, and team skills.

In this article, I want to share some of my favourite soft skill interview questions that I have come across (or used myself) during the interview process. Many of them are aimed towards Software Engineering roles, but you could adapt depending on the role that you are hiring for.

There is no right or wrong answer to these questions - they are simply there to give you inspiration during your next interview.

Questions such as:

• Why are you leaving your old workplace?
• Why are you interested in working here?
• How would your colleagues describe you if I asked them to tell me about you?
• What criticisms or strengths might they mention?
• Do you have any goals that you would like to achieve in the next few years?
• What made you want to be an [insert role here]?
• What was the last book that you read?
• Do you have any personal projects?
• What frustrates you at work?
• Tell me about a time when you had a difficult working relationship with someone at work - how did you handle it?
• When do you consider a piece of work to be finished?
• What do you think of TDD?
• Do you like it - why or why not?
• What’s the worst technical mistake that you’ve made? Or maybe the “best” outage you’ve ever been part of?
• What do you think is the best way to collaborate on a project?
• Imagine that you are in the middle of a development cycle and there is a major change in the functionality of a feature that you have been working on. How do you respond? What questions do you ask?
• What do you do when someone in your team strongly disagrees with you?How do you ensure your opinion is heard?
• How do you ensure you hear others opinions?

These are just a few questions that I like to ask and they give you a general understanding of the candidates self-awareness, passion, curiosity, temperament and approach to teamwork - just to name a few attributes!

If you are a hiring manager and you’ve never focussed on soft skills during an interview - give it a try, you never know what you might learn during your next interview.

I am always looking for ways to build faster, slicker web pages. Where applicable, I’ll use great new browser features such as service workers, HTTP/2 and improved compression, just to name a few. But what if I told you there was a way to build even faster web pages? I’d like to introduce you to the Streams API.

Before we go any further, let’s go right back to basics. What is a stream? A stream is data that is created, processed, and consumed in an incremental fashion, without ever reading all of it into memory. Streams have been available server side for years, but web streams are available to JavaScript, and you can now start processing raw data with JavaScript bit by bit as soon as it is available on the client-side, without needing to generate a buffer, string, or blob.

The benefit of using streams is that if you are sending large chunks of data over the web, you can start processing the data immediately as you receive it, without having to wait for the full download to complete. For example, if you think of a large video file and imagine how long it might take to download the whole file. Using a stream allows you to download only the small amount of data that you need to view the video instead of the whole file - this means that you can view the video as quickly as the network takes to get you just those bytes.

Streams also come with many other benefits, one of them being that they reduce the amount of memory that a large resource takes up. For example, if we needed to download a large file, process it, and keep it in memory, this could become a problem. However, with streaming, we can reduce the amount of memory that a large resource takes up because we are processing the data piece by piece; this feature is known as flow control and plays an important role in web streams.

At this point, you might be a little skeptical about using streams - but let me do my best to convince you of their great advantages. In this article, I am going to download and process a large JSON file using the Streams API and instantly write the data to a web page as we receive it, instead of when all of the data is downloaded.

What is NDJSON?

Before we go any further, it’s worth diving into a data format called NDJSON that we will be using throughout this example. If you haven’t heard of NDJSON before, it is a type of JSON called Newline delimited JSON. Wait...not another data format!? I totally agree with you, but NDJSON is needed because there is currently no standard for transporting instances of JSON text within a stream protocol. NDJSON looks very similar to JSON, the only difference is that each new line contains a new record, which allows us to stream and process one record at a time. If we sent traditional JSON in a stream, and processed it in chunks, it wouldn’t come through as valid JSON. This is why NDJSON is perfect for this situation.

Let’s look at a simple example comparing the two formats. Imagine the following JSON file:

[
    {"id":1,"name":"Alice"},
    {"id":2,"name":"Bob"},
    {"id":3,"name":"Carol"}
]

The exact same data expressed as NDJSON looks like this:

{"id":1,"name":"Alice"}
{"id":2,"name":"Bob"}
{"id":3,"name":"Carol"}

If you’d like to learn more about NDJSON, I recommend reading the following article.

With this in mind, let’s get started!

Getting Started

In order to get started, we need to get a little acquainted with the Fetch API. If you’ve used it before, you may be aware that the Fetch API exposes Response bodies as ReadableStream instances. This means that they represent a readable stream of byte data. By tapping into this ReadableStream, we are able to process data incrementally in a stream instead of buffering it all into memory and processing it in one go.

We’ll be looking at a concept called “piping” shortly which I think helps describe streams really well. Piping provides a chainable way of piping the current stream through a transform stream or any other writable/readable pair. The great thing about it is that it allows data to flow from one pipe to the next!

Let’s take a look at the following function.

/**
 * Fetch and process the stream
 */
async function process() {
    // Retrieve NDJSON from the server
    const response = await fetch('http://localhost:3000/request');

    const results = response.body
        // From bytes to text:
        .pipeThrough(new TextDecoderStream())
        // Buffer until newlines:
        .pipeThrough(splitStream('\n'))
        // Parse chunks as JSON:
        .pipeThrough(parseJSON());

    // Loop through the results and write to the DOM
    writeToDOM(results.getReader());
}

In the code above, we started off by making a request to a local server for a NDJSON file. The body of the response is available as a readable stream which, as the name implies, allows us to read data out of it.

Using the pipeThrough() method, we can pipe the data we received through to another type of stream (Writeable, Transform, etc.) and process it accordingly. In the code above, I’ve piped the body of the response through to another function called splitStream() and then again to another one called parseJSON(). Firstly, splitStream() takes the result of the NDJSON file and splits it based on each new line - making this the perfect format for streaming! Next, parseJSON() takes each chunk of data and parses the JSON to ensure that it is valid.

Now that our stream is ready to use, we can write the data to the page. With the readableStream, we can use getReader() to create a reader that locks the stream to the new reader. While the stream is locked, no other reader can be acquired until this one is released. This functionality is especially useful for creating abstractions that desire the ability to consume a stream in its entirety. By gett)ing a reader for the stream, you can ensure nobody else can interleave reads with yours or cancel the stream, which would interfere with your abstraction.

We can then iterate through the results in the reader.

/**
 * Read through the results and write to the DOM
 * @param {object} reader 
 */
function writeToDOM(reader) {
    reader.read().then(
        ({ value, done }) => {
            if (done) {
                console.log("The stream was already closed!");

            } else {
                // Build up the values
                let result = document.createElement('div');
                result.innerHTML = `<div>ID: ${value.id} - Phone: ${value.phone} - Result: ${value.result}</div><br>`;

                // Prepend to the target
                targetDiv.insertBefore(result, targetDiv.firstChild);

                // Recursively call
                writeToDOM(reader);
            }
        },
        e => console.error("The stream became errored and cannot be read from!", e)
    );
}

In the code above, I am iterating through each result in the reader and prepending the results to a DIV on the page. In a real world example, you might want to display your results differently, but this gives you an example of what is capable with streams!

Using the Developer Tools in Google Chrome, let’s inspect the results with streaming in place.

In the animation above, you can see that as I reload the page, the results are instantly displayed thanks to the streaming. Even though the rest of the HTTP request is still being downloaded and chunked, we are still able to process and display the results. Without streaming, we would have to wait the full 8 seconds for the file to download, and then display the results. This is much smoother!

Browser Support

Can I Use streams? Data on support for the streams feature across the major browsers from caniuse.com.

Summary

I hope I’ve managed to convince you how awesome streams are on the web. There are some great resources online, that I thoroughly recommend reading. It’s worth mentioning that you wouldn’t want to use streams in every situation, but where applicable they can make a big difference to the performance of your web application.

Web streams allow you to stream data to your users, allowing the browser to process data piece by piece as it is downloaded. Without streaming, we need to wait for the entire contents of a download to complete before we return a response, but by streaming the data instead, we can return the results of the download and process it piece by piece, allowing us to render something onto the screen even sooner. Faster web pages = happier users!

If you’d like to see a working example of this code, please over to Github to find out more.

Have you ever thought about building a Progressive Web App or even adding a service worker to your website? Perhaps you’ve considered it, but weren’t too sure about whether or not your users were on browsers that supported these features. Would they even benefit from their features? Speaking from experience, I know how challenging it can be to convince key stakeholders about trying out new technology on your product.

Without a doubt, one of the best ways to go about this is to start tracking usage and check the results. It's much easier to convince someone about a great piece of tech, if you are able to show the benefit to the users.

In this article, I am going to run through a small snippet of code that you can use to track service worker support on your site. We'll be pushing the results through to Google Analytics, but this could be tailored to suit any Web analytics package.

Tracking Service Worker Support

On this very blog, I am using Google Analytics as a web analytics tool. It allows me to easily see useful information about how my site is performing and lets me track page views. It also has the ability to track certain events, which works perfectly for this use case.

I won’t go into too much detail about how to set up Google Analytics on your site, as the online documentation does a great job of this. However, let's take a look at an empty HTML page with the basic Google Analytics code in place.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>Tracking Service Worker Support</title>
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-xxx-1"></script>
<script>

     
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

    
  gtag('config', 'UA-xxx-1');
</script>
  </head>
  <body>
  
  </body>
</html>

According to the documentation, the code above is typical setup for Google Analytics. In order for us to start using this, we need to add a check to determine service worker support and push it through to Google Analytics.

I added the following code to each page just after the closing body tag.

<script>
function getServiceWorkerSupport() {
  if ('serviceWorker' in navigator) {
    return 'supported';
  } else {
    return 'unsupported';
  }
}


// Push the result to Google Analytics
gtag('event', 'service-worker-support', {
      'event_category' : 'service-worker-support',
      'event_label' : getServiceWorkerSupport()
    })
</script>

Let’s break this down. I've created a function called getServiceWorkerSupport() which uses feature detection to determine if the current browser is capable of using service workers.

For each visit to my site, this information will get pushed through to Google Analytics. For this blog, I wanted to take things a little further and determine exactly how many pages were actually controlled by an active service worker. For example, if someone visits a site with a service worker on it, it will get installed the first time and only used the next time they reload the page or navigate away. I wanted to determine how many people were both supported and using my service worker.

With this in mind, I updated the function above to include the condition below.

function getServiceWorkerSupport() {
  if ('serviceWorker' in navigator) {
    return navigator.serviceWorker.controller ? 'controlled' : 'supported';
  } else {
    return 'unsupported';
  }
}

After a few days of tracking data, I logged into the Google Analytics dashboard and checked the numbers. Start by expanding the menu and navigate to Behavior -> Events -> Top Events and you should see something a little similar to the image below.

From there, choose the Event label as the primary dimension and show as a chart. I’ve tried to outline the image below (in red) to give you a better idea of this in action.

In the image above, the blue (83.8%) represents all visits to the site that support service workers. The green (10.5%) represents of all visits to my site were controlled by a service worker and served cached data. Finally, the orange shows that around 3% of all visits to my site were from browsers that don’t support service workers.

That is a big portion of my users that not only have browser support for service workers, but also already benefit from the service worker on my site!

Summary

It's worth mentioning that a big portion of my users are tech savvy and interested in learning about new browser technology. Depending on your target audience, you might find that your numbers are completely different.

If you find that the majority of your users would benefit from converting your site to a Progressive Web App, or even a service worker, it might be worth considering it. Who knows…. you might even find that service workers are good for the environment!

After a long overdue refresh, I have recently moved this blog to Ghost CMS. Overall, I have been very impressed. Ghost is easy to use as an editor, quick to make updates, and the Ghost team are constantly pushing out updates - which is great considering that it is completely open source.

In order to get started with Ghost, I currently user the AWS free tier and run this blog on a Micro instance. Considering this is the smallest instance available, I have been very impressed with the performance of this site. I also chose to go with the Bitnami Ghost stack for AWS in order to get up and running as quickly as possible.

While Ghost has been ticking away nicely, I have recently been having some serious issues with my Amazon EC2 instance and I wanted to write this article to help any others that might be experiencing the same problem! I’d also like to refer back to this in case it happens to me again.

AWS Instance scheduled for Retirement

As far as I was concerned, my blog was up and running and everything was working as expected. After about 3 months of running undisturbed, I received a warning from AWS that my “instance was scheduled for retirement due to underlying hardware degradation and that it may already be unreachable”. What!?

Sure enough, I logged in to the Amazon management console and it was down. My blog was also returning a 503 Service Unavailable error. Fortunately I was able to stop the instance and restart it again - but this meant I needed to quickly backup my blog and restore it to another instance.

Wait...you don’t back your blog up!? Yeah I know….rookie mistake!

Making a backup of your Ghost blog

There are two ways to make a backup of your Ghost blog on EC2. The first option is to create an image of your EC2 instance which you can simply use to restore from at any point in time. Think of it as a “snapshot” for the given time.

If you log into the AWS console, select Instances and then select the instance you wish to back up. Next, choose Actions -> Image -> Create Image.

This will then create AMI under images that you can use to restore from. You can simply choose Launch Instance and then look under My AMIs to restore from your newly created image.

You could even take this a step further and automate the whole process, so that you have an up to date image to restore from at any point. Using a Cloudwatch event, you can take a scheduled snapshot of your instance. I won’t go into it in this article, as this is an article in itself - but please follow the link above for more details.

The second option is to manually backup your blog using the UI on your Ghost blog. This option is ideal if you are looking to restore to a completely clean, new version of the Bitnami Ghost image. You’ll need to start by exporting all of the content on your blog. Log into your admin interface, and head over to Labs.

Choose Export your Content, and it will download all of your posts and settings in a single JSON file.

If you have any redirects or routes set up on your Ghost blog, you can do this in the same way.

We are almost done - next you’ll need to take a back up of the design of your blog. Head over to the Design tab in your admin interface.

From there, click on the download link and you’ll receive a ZIP file containing all of the assets that make up the design of your site.

The best thing about the Ghost admin interface is that it allows you to import everything back in the same way. If you have another instance up and running, log into the admin interface and upload all of the exported content that you have. Nice and easy!

Backing up the images on a Ghost blog

Now this is all good and well, but the eagle-eyed amongst you might have noticed that we haven’t actually downloaded any of the images associated with the blog. You might have restored all of the content to your new instance, but if you visit the pages on your site, you may notice that your images now point to broken links!

At the moment, the Ghost admin interface doesn’t let you download a copy of all of your images. Fortunately there is a way around this, although its a bit of a pain. All of the images for your Bitnami Ghost blog are located at ‘~/apps/ghost/htdocs/content/images’.

I switch between Windows and a Mac - so I used PSCP on a Windows machine for the command below. However, the command using SCP in linux is almost identical. I started by downloading all of the images on my blog to my local machine.

pscp -r -i C:\Users\dhume\Documents\xxx.ppk bitnami@xxx.compute-1.amazonaws.com:/home/bitnami/apps/ghost/htdocs/content/images C:\dean 

Let’s break this down a little. Firstly, I am running the PSCP command and am linking to the .ppk file with my login credentials. Putty SCP uses PPK instead of PEM, but you can easily create one using the Putty tool. Next, point to the EC2 instance that I want to copy the images from, as well as the path to the images on the EC2 instance. The last parameter points to the folder on my local machine.

After running this command, I can see that all of the images have been downloaded to my local drive.

Restoring images on a Ghost blog

So far so good! Our images have been downloaded onto the local drive but we still need to upload them to the new EC2 instance. Using a similar command to the one we used above, I now need to flip the command and point to the new EC2 instance.

pscp -r -i C:\Users\dhume\Documents\xxx.ppk C:\dean\images\2018 bitnami@xxx.us-west-2.compute.amazonaws.com:/home/bitnami/apps/ghost/htdocs/content/images/ 

If all run successfully, you should now see the images on your site!

Removing the ‘manage’ banner on a Ghost blog (Bitnami)

If you spin up a new Bitnami Ghost instance, there are still a few things that you need to do before the site is ready to go ‘live’. On a completely fresh instance, you may notice a banner in the bottom right hand corner when you view your site in a web browser.

In order to remove this, you’ll need to SSH into your instance and run the following command in the terminal.

sudo /opt/bitnami/apps/ghost/bnconfig --disable_banner 1

If you refresh the page, you’ll notice that this has now been removed.

Updating the domain name on a Ghost blog

When you spin up a new Bitnami Ghost instance on EC2, you’ll be able to access it from the default domain that AWS assigns to it. However, you’ll want to point this to a real domain name and that involves updating a config setting in order for Ghost to link correctly.

In order to do this, you’ll need to SSH into your instance and run the following command.

sudo /opt/bitnami/apps/ghost/bnconfig --machine_hostname yourdomainname.com

It takes a little while to finish running, but this command will update your domain accordingly. Provided you’ve updated your DNS details with your hosting provider - when you visit the domain name on your website - all links will now be in working order.

I have sometimes found that the command above appends a :80 port number to the end of the domain name. This can mess with RSS feeds and links. In that case, you can always directly edit the values in the config.production.json file using the following command:

sudo nano /opt/bitnami/apps/ghost/htdocs/config.production.json

Updating a Ghost blog to the latest version

Considering Ghost is completely open source and built largely by the community, it’s pretty impressive to see the number of updates that they constantly ship. If you log into the admin section of your Ghost blog, and click on About - you should notice something a little like the image below.

The Ghost admin interface intelligently lets you know that you are on an older version and what new features are available if you update. Not surprisingly, updating it easier than you think.

Start off by SSH’ing into your instance and navigate to /apps/ghost/htdocs. Next, you’ll need to run the following command.

sudo ghost update

If everything runs successfully, you’ll have the latest version of Ghost up and running!

Stopping and Starting Ghost again

I have noticed a few instances where you suddenly encounter a 503 error when you are running an Amazon Micro instance. This could be caused due to having low memory on the instance and Ghost struggles with this. I find that a quick stop and start of Ghost should sort this problem out too.

sudo /opt/bitnami/ctlscript.sh stop

sudo /opt/bitnami/ctlscript.sh start

Summary

That’s about it. I hope that this article has been helpful to you. As I experiment with Ghost and make changes to this blog, I will be sure to keep it updated. Hopefully this serves to help you, but as a reminder for myself in case this happens again!

Whether you lead a small team of developers, a tech startup or a huge department, it’s important that you constantly strive to improve the processes and tools that you use. If things are going really well, it’s easy to become confident in your abilities and overlook the need to self assess where you are at as a team.

One of my favourite tools for doing this is a Maturity Matrix. The idea behind this matrix is that you have a list of best practices and you assess your team or department accordingly. For example, imagine you were building the perfect organisation - what would it look like? What tools / processes would they use? I like to jot all of these things down in the Maturity Matrix and compare where we might be as a team versus the reality of where we are at. It’s a bit like the Joel test but on a much larger scale.

Let’s take the spreadsheet below as an example.

The spreadsheet above contains a list of Software Engineering best practices for a department (click here to download if you can’t see above). Sure, this list might not contain every single best practice available and is somewhat customised to my experience - but it's a good start and gives an insight into the things that could be improved upon. The best thing about this is that you can tailor it to suit your own departments needs and regularly reassess where your team is at.

So, how do you use this? Well, I like to use a Red / Amber / Green rating to compare where things are. Red is for non existent. Amber is for work in progress, or something we partially do. Finally Green is for something that we work on.

It’s important to be honest with yourself and rate yourself accordingly - after all, this is only to help improve things for you and your team! Once you’ve completed the Maturity Matrix you should start to get a bigger picture of the things that you need to work on. For example, if there are a lot of reds - then perhaps those are the more immediate things that you need to work on.

It’s important to not only look at Software Engineering best practices - but also People best practices. Below is another Maturity Matrix that I like to use to assess how things are in terms of people.

Again, this list is something that I like to use and have personally built up - but it might not be suited to you - it’s up to you to adjust and add accordingly.

Summary

As a team, it’s important to constantly look for new avenues to improve. I know that in my own experience, I can sometimes become lazy and neglect some of the important things that should be done. This is why I like using a Maturity Matrix like this to keep me and my teams honest.