Implement multi-threading with .NET runspaces in Powershell
There are multiple options on how to implement multi-threading in Powershell. They are all well known to engineers:
- Powershell Jobs
- Powershell Workflows
But there is another one, which is not quite popular (because of it's complexity), but very powerfull: .NET runspaces. While it's quite difficult to implement, it don't have main disadvantage of native Powershell ways - we will not spawn tons of powershell.exe processes. All work will be done within a single process and that will highly increase overall script performance.
I will not tell you about theory (mainly because I don't clearly understand details), but will give you some insights and script templates to start moving.
What I know - is that runspace is the single space to invoke some code, while runspacepool - is the pool for the multiple runspaces and .NET knows how to aumatically manage them.
So, let's start.
initialSessionState will hold typeDatas and functions that will be passed to every runspace.
$initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault();
Now we'll define some function.
function getSomeThing { param ( [Parameter(Mandatory=$true)][string]$url ); try { $doc = Invoke-WebRequest $url -Method Get -DisableKeepAlive -ErrorAction SilentlyContinue; } catch [System.Net.WebException] { Write-Host "$url : Invoke-WebRequest Error"; return $false; } return $doc; } $getSomeThing_def = Get-Content Function:\getSomeThing; $getSomeThing_SessionStateFunction = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList 'getSomeThing', $getSomeThing_def; $initialSessionState.Commands.Add($getSomeThing_SessionStateFunction);
In last three strings we're adding our function to the initialSessionState. Ok, maybe you like object-oriented approach as I do, then here's how you'll define your TypeData and add it to session state:
$init = @{ MemberName = 'Init'; MemberType = 'ScriptMethod'; Value = { Add-Member -InputObject $this -MemberType NoteProperty -Name Url -Value $null; Add-Member -InputObject $this -MemberType NoteProperty -Name Title -Value $null; }; Force = $true; } $populate = @{ MemberName = 'Populate'; MemberType = 'ScriptMethod'; Value = { param ( [Parameter(Mandatory=$true)][string]$url ); $this.url = $url; $this.title = getSomeThing($url); }; Force = $true; } Update-TypeData -TypeName 'Custom.Object' @Init; Update-TypeData -TypeName 'Custom.Object' @Populate; $customObject_typeEntry = New-Object System.Management.Automation.Runspaces.SessionStateTypeEntry -ArgumentList $(Get-TypeData Custom.Object), $false; $initialSessionState.Types.Add($customObject_typeEntry);
Next we'll define our main, entry point to runspace.
$ScriptBlock = { Param ( [PSCustomObject]$url ) $page = [PsCustomObject]@{PsTypeName ='Custom.Object'}; $page.Init(); $page.Populate($url); $Result = New-Object PSObject -Property @{ title = $null url = $url }; return $Result; }
And - finally - we're going to spin things up.
$Throttle = 15; #threads $RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, $Throttle, $initialSessionState, $Host); $RunspacePool.Open(); $Jobs = @(); $i = 0; foreach($url in $urls) { #$urls - some array of URLs $i++; $Job = [powershell]::Create().AddScript($ScriptBlock).AddArgument($url); $Job.RunspacePool = $RunspacePool; $Jobs += New-Object PSObject -Property @{ RunNum = $i; Pipe = $Job; Result = $Job.BeginInvoke(); } } $results = @(); foreach ($Job in $Jobs) { $Results += $Job.Pipe.EndInvoke($Job.Result); }
Results are collected in the very end.
That's it.
Tags: powershell (en)