Quick tip: PowerShell performance

It was the best of times, it was the worst of times…

I generally like doing scripting with PowerShell. Scripting has a certain airiness, or cloudiness if you want. You mostly need your script for a short period of time, and don’t have to waste time trying to think how you can write it better in such a way that the developers of tomorrow can still understand it by looking at your lines. It’s just got a job to do, it does it, and then it’s done.

But you still need to write good code. Otherwise you sometimes get stuck on these annoying issue. Like what happened with me some time ago. I had a script that needed to compare a lot of files from several sources and write results somewhere else. I had tried it out on a small set of data and it went fine. But then I executed the script on a few hundreds of thousands of rows and it just clogged down. It literally ran like 24 hours before it was ready.

After googling some, and sparring with a colleague, we came up with a few changes that effectively reduced the time to 1,5 hours. The difference was due to the following changes:

I stopped using arrays to store rows of data

The usual way I would create an array and add to it, was as follows:

# Create an array
$array = @()

# Add an item to it
$array += $item

This shorthand form turned out to be very bad for performance. Below the surface it’s actually copying the entire array every time you push an item. You can imagine how this will work out if you have 300.000 items in it.

A better solution is to define an arraylist and add items to it using it’s Add() function:

# Create an array list
$arrayList = [System.Collections.ArrayList]@()

# Add an item to it.
$arrayList.Add($item)     

This method is a lot (!) faster on large arrays.

I started using Hash Tables for comparing rows of data

Now querying in an array list of 300.000 rows data using Where-Object can also be slow. In some situations you might consider using a hash table. The idea of a hash table is that it saves key-value pairs. Adding to and looking up in a hash table and even iterating through it really is lightning fast. So if you have a useful key and you do not need extensive where-statements, this is the way to go.

A hash table can be used as follows:

# create the hash table
$hashtable = @{} 

# Store a value to the table
$hashtable["sometextvalueaskey"] = [pscustomobj] {}

# Get a value from the table
$item = $hashtable["sometextvalueaskey"]

# Iterating over a hash table
foreach ($keyValue in $hashtable.GetEnumerator()) {
    # Do stuff
}                  

I minimised on logging

The last tip for today is logging. And I do a lot of it. I regularly use Start-Transcript and Stop-Transcript commandlets to save logs to log-files, especially for long running scripts.

Start-Transcript "$PWD/SomeLogFile.log"

# Do a lot of scripting

Stop-Transcript

The reason is that I want to be able to see what’s actually happened when I reboot my laptop the next morning. The drawback, and that might seem like an open door, is that I/O operations take significant time. In my search for control I sometimes log too much. And that bogs down on these big scripts.

So I had to change my attitude. I went Marie Kondo on the logging. With every line I checked: 'Does this spark joy?'. In other words: I still do logging, but I try not to output too much, unless when debugging and trying out scripts.

Conclusion

You can imagine these speedbumps sparked a lot of joy. I’m very interested in knowing more of these time saving gems. So if you have any, please let me know.

Sources

• About PowerShell and arrays