Thumbnail Quick tip: PowerShell performance

Quick tip: PowerShell performance

It was the best of times, it was the worst of times…

I generally like doing scripting with PowerShell. Scripting has a certain airiness, or cloudiness if you want. You mostly need your script for a short period of time, and don’t have to waste time trying to think how you can write it better in such a way that the developers of tomorrow can still understand it by looking at your lines. It’s just got a job to do, it does it, and then it’s done.

But you still need to write good code. Otherwise you sometimes get stuck on these annoying issue. Like what happened with me some time ago. I had a script that needed to compare a lot of files from several sources and write results somewhere else. I had tried it out on a small set of data and it went fine. But then I executed the script on a few hundreds of thousands of rows and it just clogged down. It literally ran like 24 hours before it was ready.

After googling some, and sparring with a colleague, we came up with a few changes that effectively reduced the time to 1,5 hours. The difference was due to the following changes:

I stopped using arrays to store rows of data

The usual way I would create an array and add to it, was as follows:


    # Create an array
    $array = @()

    # Add an item to it
    $array += $item
     

This shorthand form turned out to be very bad for performance. Below the surface it’s actually copying the entire array every time you push an item. You can imagine how this will work out if you have 300.000 items in it.

A better solution is to define an arraylist and add items to it using it’s Add() function:


    # Create an array list
    $arrayList = [System.Collections.ArrayList]@()

    # Add an item to it.
    $arrayList.Add($item)     
     

This method is a lot (!) faster on large arrays.

I started using Hash Tables for comparing rows of data

Now querying in an array list of 300.000 rows data using Where-Object can also be slow. In some situations you might consider using a hash table. The idea of a hash table is that it saves key-value pairs. Adding to and looking up in a hash table and even iterating through it really is lightning fast. So if you have a useful key and you do not need extensive where-statements, this is the way to go.

A hash table can be used as follows:


    # create the hash table
    $hashtable = @{} 

    # Store a value to the table
    $hashtable["sometextvalueaskey"] = [pscustomobj] {}

    # Get a value from the table
    $item = $hashtable["sometextvalueaskey"]

    # Iterating over a hash table
    foreach ($keyValue in $hashtable.GetEnumerator()) {
        # Do stuff
    }                  
     

I minimised on logging

The last tip for today is logging. And I do a lot of it. I regularly use Start-Transcript and Stop-Transcript commandlets to save logs to log-files, especially for long running scripts.


    Start-Transcript "$PWD/SomeLogFile.log"

    # Do a lot of scripting

    Stop-Transcript
     

The reason is that I want to be able to see what’s actually happened when I reboot my laptop the next morning. The drawback, and that might seem like an open door, is that I/O operations take significant time. In my search for control I sometimes log too much. And that bogs down on these big scripts.

So I had to change my attitude. I went Marie Kondo on the logging. With every line I checked: 'Does this spark joy?'. In other words: I still do logging, but I try not to output too much, unless when debugging and trying out scripts.

Marie Kondo: Does this spark joy

Conclusion

You can imagine these speedbumps sparked a lot of joy. I’m very interested in knowing more of these time saving gems. So if you have any, please let me know.

Sources


powershell performance tips
Support me by sharing this

Comments

More

More blogs

Quick tip: how to deploy to web app subfolders
Quick tip: how to deploy to web app subfolders

A quick tip on the possibilities of deploying to web app subfolders in Azure DevOps pipelines.

Read more
Fixing the Nginx trailing slash issue on Linux based Azure Web Apps
Fixing the Nginx trailing slash issue on Linux based Azure Web Apps

When running a Linux based Azure Web App with the PHP 8 runtime stack, URL's without trailing slashes pose a problem.

Read more
Creating a beautiful documentation site with MkDocs
Creating a beautiful documentation site with MkDocs

MkDocs is a great tool to create straightforward documentation sites. It's 1 of 2 static site generators I've recently worked with.

Read more

Thanks

Thanks for reading

Thanks for reading my blog, I hope you got what you came for. Blogs of others have been super important during my work. This site is me returning the favor. If you read anything you do not understand because I failed to clarify it enough, please drop me a post using my socials or the contact form.


Warm regards,
Martin

Microsoft 365 Architect
I4-YOU Business Solutions b.v.