Gapski

PowerShell scripts can grow very slow when you (a) need to process a lot of items and (b) you are using the PowerShell pipeline. Let’s today find out why that is, and what you can do about it.

To visualize the underlying problem, let’s first create a test case that shows how PowerShell gets slowed down considerably. For this, we need a lot of items, so the code below generates a list of all files in your Windows folder, which can take a couple of seconds to generate.

# get large data sets
$files = Get-ChildItem -Path c:windows -File -Recurse -ErrorAction SilentlyContinue
$files.Count

Let’s send the files into a pipeline, and pick only files larger than 1MB. In the examples below, we are piping the content of $file solely for the purpose to have reproducible input data. In real life, you’d of course never use variables and instead stream results directly from commands.

Measure-Command {
    $largeFiles = $files | Where-Object { $_.Length -gt 1MB }
}
$largeFiles.Count

In our test, the code took between 3 and 4 seconds and produced 3485 “large” files. Results may vary on your machine.

Where-Object really is just a ForEach-Object with an If-Clause inside of it, so let’s try and replace Where-Object by If:

Measure-Command {
$largeFiles = $Files | ForEach-Object {
        if ($_.Length -gt 1MB)
        { $_ }
    }
}
$largeFiles.Count

The result is the same, yet the time is cut in half.

ForEach-Object is really just an anonymous script block with a process block, so next, try this:

Measure-Command {
$largeFiles = $Files | & {
        process
        {
            if ($_.Length -gt 1MB)
            { $_ }
        }
    }
}
$largeFiles.Count

Again, the result is the same, but the time was cut down from an initial 4 seconds to roughly 100 milliseconds (factor 40).

As it turns out, when you feed data to commands via pipeline, PowerShell invokes the parameter binder for each transmitted item, and this can add up to significant delays. Since both ForEach-Object and Where-Object use parameters, the binder is activated.

When you instead use anonymous script blocks with a process block inside but with no parameters, you bypass all of the parameter binding and can speed up PowerShell pipeline operations to a degree that makes a difference even in the wild.

---

psconf.eu – PowerShell Conference EU 2019 – June 4-7, Hannover Germany – visit www.psconf.eu There aren’t too many trainings around for experienced PowerShell scripters where you really still learn something new. But there’s one place you don’t want to miss: PowerShell Conference EU – with 40 renown international speakers including PowerShell team members and MVPs, plus 350 professional and creative PowerShell scripters. Registration is open at www.psconf.eu, and the full 3-track 4-days agenda becomes available soon. Once a year it’s just a smart move to come together, update know-how, learn about security and mitigations, and bring home fresh ideas and authoritative guidance. We’d sure love to see and hear from you!

ReTweet this Tip!

I’ve just finished writing another book on PowerShell . The book looked at a number of core Windows Features and components, from AD, DHCP/DNS, SMB file sharing and FSRM, Hyper, V and more. Having gone through the process of witting over 125 scripts… Read the full text.

In testing any new feature, one technique for getting users to use (and test) the feature is known as feature flags . Essentially these are settings (flags) that signal you should gain access to those experimental features. The allows a user to opt-in… Read the full text.

Where-Object is one of the most frequently used PowerShell commands, yet it looks unfamiliar to new PowerShell users. Those who are familiar with the “SQL” database query language can use Where-Object like the Where statement in SQL clauses: it is a client-side filter that eliminates unwanted items. This line would take all services and display only those who are actually running:

Get-Service | Where-Object { $_.Status -eq "Running" }

To better understand how Where-Object works, it is basically an If-Clause for the pipeline. The above code is the equivalent of this:

Get-Service | ForEach-Object {
    if ($_.Status -eq 'Running')
    { $_ }
}

Or, in a completely pipeline-free “traditional” approach:

$services = Get-Service
Foreach ($_ in $services)
{
  if ($_.Status -eq 'Running')
  {
    $_
  }
}

---

psconf.eu – PowerShell Conference EU 2019 – June 4-7, Hannover Germany – visit www.psconf.eu There aren’t too many trainings around for experienced PowerShell scripters where you really still learn something new. But there’s one place you don’t want to miss: PowerShell Conference EU – with 40 renown international speakers including PowerShell team members and MVPs, plus 350 professional and creative PowerShell scripters. Registration is open at www.psconf.eu, and the full 3-track 4-days agenda becomes available soon. Once a year it’s just a smart move to come together, update know-how, learn about security and mitigations, and bring home fresh ideas and authoritative guidance. We’d sure love to see and hear from you!

ReTweet this Tip!

When you define mandatory parameters in PowerShell, the user gets prompted for the value if it is missing. The prompt uses the parameter name only as you can see when you run this code:

param
(
  [Parameter(Mandatory)]
  [string]
  $UserName

)

"You entered $Username"

 
UserName: tobi
You entered tobi 
 

To get more descriptive prompts, you can use more explicit variable names:

param
(
  [Parameter(Mandatory)]
  [string]
  ${Please provide a user name}

)

$username = ${Please provide a user name}
"You entered $Username"

 
Please provide a user name: tobi
You entered tobi  
 

Simply wrap the param() structure in a function to produce commands:

function New-CorporateUser
{
    param
    (
      [Parameter(Mandatory)]
      [string]
      ${Please provide a user name}
    )

    $username = ${Please provide a user name}
    "You entered $Username"
}

 
PS C:> New-CorporateUser
Cmdlet New-CorporateUser at command pipeline position 1
Supply values for the following parameters:
Please provide a user name: Tobi
You entered Tobi 
 

The flip side is that blanks and special characters in parameter names make it impossible to assign values via command line because parameters cannot be quoted:

 
PS C:> New-CorporateUser -Please  provide a user name
 

---

psconf.eu – PowerShell Conference EU 2019 – June 4-7, Hannover Germany – visit www.psconf.eu There aren’t too many trainings around for experienced PowerShell scripters where you really still learn something new. But there’s one place you don’t want to miss: PowerShell Conference EU – with 40 renown international speakers including PowerShell team members and MVPs, plus 350 professional and creative PowerShell scripters. Registration is open at www.psconf.eu, and the full 3-track 4-days agenda becomes available soon. Once a year it’s just a smart move to come together, update know-how, learn about security and mitigations, and bring home fresh ideas and authoritative guidance. We’d sure love to see and hear from you!

ReTweet this Tip!

Not all PowerShell cmdlets ship with PowerShell. Many are part of 3rd party modules which in turn ship when you install certain software, or use certain Windows versions.

In the previous part we produced a function that dumps all external commands found in a script. With just a little extra effort, this can be turned into a useful compatibility report: any cmdlet originates from a module, and any module with a name starting with “Microsoft.PowerShell” ships with PowerShell. Any other modules are specific to certain Windows versions or 3rd party extensions.

Check out this function:

function Get-ExternalCommand
{
    param
    (
        [Parameter(Mandatory)][string]
        $Path
    )
    function Get-ContainedCommand
    {
        param
        (
            [Parameter(Mandatory)][string]
            $Path,

            [string][ValidateSet('FunctionDefinition','Command')]
            $ItemType
        )

        $Token = $Err = $null
        $ast = [Management.Automation.Language.Parser]::ParseFile($Path, [ref] $Token, [ref] $Err)

        $ast.FindAll({ $args[0].GetType().Name -eq "${ItemType}Ast" }, $true)

    }


$functionNames = Get-ContainedCommand $Path -ItemType FunctionDefinition | 
  Select-Object -ExpandProperty Name
   
    $commands = Get-ContainedCommand $Path -ItemType Command 
    $commands | Where-Object {
      $commandName = $_.CommandElements[0].Extent.Text
      $commandName -notin $functionNames
      } | 
      ForEach-Object { $_.GetCommandName() } |
      Sort-Object -Unique |
      ForEach-Object { 
    $module = (Get-Command -name $_).Source
    $builtIn = $module -like 'Microsoft.PowerShell.*'

    [PSCustomObject]@{
        Command = $_
        BuiltIn = $builtIn
        Module = $module
    }
    }
}

Here is a sample report from a PowerShell script listing all external cmdlets and whether they are part of PowerShell or come from external modules:

 
PS> Get-ExternalCommand -Path $Path


Command                BuiltIn Module                         
-------                ------- ------                         
ConvertFrom-StringData    True Microsoft.PowerShell.Utility   
Get-Acl                   True Microsoft.PowerShell.Security  
Get-ItemProperty          True Microsoft.PowerShell.Management
Get-Service               True Microsoft.PowerShell.Management
Get-WmiObject             True Microsoft.PowerShell.Management
New-Object                True Microsoft.PowerShell.Utility   
out-default               True Microsoft.PowerShell.Core      
Test-Path                 True Microsoft.PowerShell.Management
Where-Object              True Microsoft.PowerShell.Core      
write-host                True Microsoft.PowerShell.Utility   
 
PS> 
 

---

psconf.eu – PowerShell Conference EU 2019 – June 4-7, Hannover Germany – visit www.psconf.eu There aren’t too many trainings around for experienced PowerShell scripters where you really still learn something new. But there’s one place you don’t want to miss: PowerShell Conference EU – with 40 renown international speakers including PowerShell team members and MVPs, plus 350 professional and creative PowerShell scripters. Registration is open at www.psconf.eu, and the full 3-track 4-days agenda becomes available soon. Once a year it’s just a smart move to come together, update know-how, learn about security and mitigations, and bring home fresh ideas and authoritative guidance. We’d sure love to see and hear from you!

ReTweet this Tip!