Jobs, and how they save your life

No. I don't mean your job. You are reading this page because you have a job and you need your life to be easier, or you don't have a job, and cats got boring. What I do know is Powershell, and it's ability to thread workloads, has made me keep my current job, as well as other important things like sanity, clean underpants, hair, carpel-tunnel free hands. You get the idea. It's also a topic that when I was starting out learning about jobs, I had trouble finding many good, easy to use examples of how I could apply them for what I need.

There are many types of jobs in Powershell, and just as many different ways to execute them. However, for the most part, you either have a script you wish you could thread, or you have the requirement to thread a lot of things instead of running them sequentially. A few instances this might be used in is, for example, starting or stopping services on hundreds of servers, copying a file to a remote file system on each server, creating/nesting groups based off of a document in AD, grabbing log files from many different hosts, or whatever it is you may be doing. I hope the valuable application of this has already become apparant.

Lets get something clear first. Start-Job is expensive. It also starts a seperate instance of Powershell. This brings with it two unique challenges:

1. You are going to catch your computer or server on fire (maybe)
2. You can't pass variables from the host shell so easily

Number 1 is because Start-Job creates a new instance of powershell, which needs resources. Personally, I have gotten away with running about 150 Powershell instances concurrently, which required about 12GB of available memory. Number 2 is valid. Since it is spinning up a fresh clean instance of Powershell, it won't have any of your session variables stored, so you need to pass them to the job. This would be the function of -ArgumentList. We will go over this in this post.

Pancake:

Start-Job -Name JobNameWhatever -Scriptblock {..} -ArgumentList $var,$var2

Easy enough right? This is the basic breakdown of Start-Job, and how 99% of the worlds population will be using it. You can start a job based on a remote script as well, which is similar to Invoke-Expression (We won't cover that here), you can use a very large and complicated script block (what you will most likely be doing), or you can just enter any plain old boring Powershell command, which you will put into a script block anyways. So let's take another example of actually using Start-Job in a loop. This will basically serve all of the needs I think you need to use it for.

$Names = Get-Content C:\PowershellPancakes\AuntJemima.txt
foreach ($Name in $Names)
    {
    $MapleSyrup =
        {
        Param ($name)
        New-Item -Name $Name -ItemType Directory -Path C:\PowershellPancakes
        }

Start-Job -Name $Name -Scriptblock $MapleSyrup -ArgumentList $Name
    }

This is just a simple script in a loop, where we are creating some delicious folders based off the names in the text file. You can open the loop however you want, in any way you want. This is the basic function. So imagine your $MapleSyrup scriptblock. You can do whatever you want in this script block. You can even take an existing script that takes manual input, put it into a script-block like so, and start a job based off of it. I have been doing this increasingly so, when someone asks me, can we do this, like, all at once? Why yes, client A, we can do this all at once. To maintain your full head of hair I suggest you put the job name to a variable you can identify. That way if you are running several hundred jobs you can skim through a list with Get-Job.

I want to go into detail about the following bit, because this may cause you a lot of headaches when you get started with jobs. So here it is:

Param ($name)

Were you expecting more? This just needs to be noted. This is where your parameters will get transferred into your scriptblock, which need to be used in -ArgumentList by Start-Job. If you don't, the value of $name will be null when the job starts, which could create unforseen problems, as well as just not work. Another note, variables you create inside script blocks don't need to be added to -ArgumentList or Param. They are contained. As long as you add the external variables being passed from outside of the scriptblock through Param and -ArgumentList, you are okay. Also remember, seperate params by a comma, like so,

Param ($name,$plants,$syrup)

Or, personally,

Param ($name,
       $syrup,
       $plants)

You have just one line you need to run?

Start-Job -Name Blueberries -Scriptblock {robocopy \\serverB\1 \\serverB\2}

What's nice about starting jobs is since they use another instance, you're free to continue using the parent shell for whatever you want. Some of you may have run into the situation where if you were running a script in the ISE and wanted to do something else on another script in the same ISE, you couldn't. Well since Start-Job creates its own instance of Powershell, you don't have to worry about this.

So you've started a job and it's just...running...somewhere...

Get-Job 

Though no, you can't exactly expand the operating job, and you will need to have some confidence in your own script. In another post we will go over how you can add some error handling to pass things back to your parent shell, or to a log file using different commands, including Try and Catch, which we touched on earlier.

But I have a job I need to run, through 400 objects in my CSV file that i'm looping through. I only have 4gb of available memory! Oh what do I do? First I will refer you to a previous posting, which goes very closely with this one. That would be What Do Until loops can do for you. They are connected directly. But how? Let's take our earlier example, and put some job control on it.

$Names = Get-Content C:\PowershellPancakes\AuntJemima.txt
foreach ($Name in $Names)
    {
    $MapleSyrup =
        {
        Param ($name)
        New-Item -Name $Name -ItemType Directory -Path C:\PowershellPancakes
        }

Start-Job -Name $Name $MapleSyrup -ArgumentList $Name
DO {
    Start-Sleep -Seconds 5
    #Give it some time
    Remove-Job -State Completed
    } Until (@(Get-Job).Count -le 5)
    }

What just happened? Well, we just limited the number of active jobs at any given time. I probably don't need to tell you why you want to do this. Especially if you are working with just huge bits of data and lots of jobs need to be created to do it. Now you can create jobs that create sub-jobs, but that's just a bit unnecessary in my personal opinion, unless each job has a lot of items it needs to complete, that would be better suited by threading those as well. Just remember your hardware limit so you don't do anything too crazy and bring everything to a halt. Also consider your network/storage/hardware or whatever it is you will be working with. Let's not be reckless here.

I think that basically covers it for Jobs. An absolutely essential function of Powershell that I think could sometimes be misunderstood. Mess around with the examples in this post and you should get a very good feel for how to incorporate these Pancakes into your own recipes. 

What Do Until loops do for you

Lately I've been getting into a lot of multi-processing and threading features of powershell. Of course, being the Powershell Pancaker, I don't typically like most of the examples I see - as oftentimes they are just too bloated, and they hurt to read (variables in the ISE burn my retinas). But I really needed this. Almost as bad as I need coffee, or excercise, or underpants.

Have you ever wanted something to happen, until it happened correctly? What if you had a bit of code, that depends on something external, maybe you want to run a script, and then (for whatever reason) go manually complete a task, and the script will continue when you are done? I don't know why you would do that. But this can be particularly useful in scenarios where maybe you are waiting for some bit of AD replication, as I was, which prompted me to write this. This is essential if you are running jobs in Powershell.

The lazy mans way out of this predicament is to use the Start-Sleep cmdlet. Which basically means you have way too much free time to run this script, and obviously you really just don't care about pretty much anything. But this doesn't work if you have a deadline, and you need this to be done as fast as possible. And doubly so if you are running your function as a seperate job, where you either arn't particularly bothered to report results from each job or - it is just unnecessary. This also doesn't work if you need to execute 400 instances of Powershell and you only have 16GB of RAM. This could be troublesome if your lazy Start-Sleep timer wasn't long enough, and unbeknownst to you, your script failed or didn't meet your success criterion.

So let's get started.

Pancake:

DO { #Start a DO loop
    $a++
    Write-Host $a
    } Until ($a -eq 10) #Write $a++ until -eq 10

This is the basic breakdown. Until (...) accepts all sorts of input. This command is useful in a number of scenarios. Inside of the DO loop, you can do pretty much anything you want. Let's try something completely non-sensible.

DO {
    Start-Sleep 2
    Write-Host Waiting for folder..
    $Folder = Test-Path C:\PowerShellPancakes\Testdir
    } Until ($Folder -eq "True")

In the above, we are using one of the unlikely scenarios that you want to manually create a folder to return some sort of output. Let's say we need to wait until this folder is created, and then when it is, copy something into it? If you are interleaving copy jobs maybe you will find this useful. Otherwise no one else will ever use this. But these are Pancakes, remember? We can put whatever we want on them. To do this we just have to add whatever we want at the end of the loop to continue on with our script.

DO {
    Start-Sleep 2
    Write-Host Waiting for folder..
    $Folder = Test-Path C:\PowerShellPancakes\Testdir
    } Until ($Folder -eq "True")
Copy-Item -Path C:\PowershellPancakes\AuntJemima.txt -Destination C:\PowershellPancakes\testdir
#Script continues as normal

As you can imagine, we can use this bit of code to transform basically anything that requires a wait time, success criteria, value limit, into something of a more fire and forget system. Is there anything else we can do with this? Hmm, what about Try and Catch? This is what you need to know if you don't want a bunch of error output while you are waiting for your criterion to become true, or, whatever it is you are doing. Let's use the following example.

DO {
    Start-Sleep 2
    Write-Host Waiting for folder..
    $Folder = Move-Item C:\PowershellPancakes\testdir -Destination C:\PowershellPancakes\Destination
        } Until ($Folder -eq "True")

This is going to be super annoying. You haven't even created TestDir yet, how are we going to move it if it doesn't exist? It's okay. Powershell will definitely let you know that it doesn't exist. Every two seconds, actually. What if you aren't as stupid as it assumes you are, and you want it to chill out until you have created this folder so it can be moved. (Again. Let's forget practicality for a moment.) Well, this is where catch and try come into scope. With try, we can do just that - try things. There is a big difference between try and do, except in Powershell, we do try. (See what I did there?) So let's look at something that won't hurt our eyes quite so much.

$ErrorActionPreference = "Stop"
DO {
    Start-Sleep 1
    Write-Host Waiting for folder..
    try {
        $GetFolder = Get-ChildItem C:\PowershellPancakes\Testdir
        }
    catch { }
        } Until ($GetFolder -eq $null)
Copy-Item -Path C:\PowershellPancakes\AuntJemima.txt -Destination C:\PowershellPancakes\testdir

Ah, much better. Thank you for understanding Powershell, don't be so impatient. There is one important thing to note, however - try and catch work great, however only for terminating errors. The above is an example of a non-terminating error. The script will still go on. So that is why we need to set our $ErrorActionPreference variable. In the above we are doing exactly what we want, in the do loop we are going to try and get the contents of that folder, and we are going to keep trying every two seconds. However, with catch, we are capturing what comes out of that try statement. You can enter a lot of things in here, you can enter some additional processing, maybe a nice Write-Error that looks better, put it out to a log file, whatever - the possibilities here are endless. You can even use the system classes to catch particular types of errors from try, if you happen to be running more than one set of commands. When I write scripts I usually leave this blank as I haven't had a reason to fill in the catch block yet, I use it to just ignore the error output. But much like If and Else statements, they follow eachother naturally.

That's all for now - this should get you started with the Do Until, Try and Catch statements.